[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:02:53 dist-test-slave-nmsz krb5kdc[8832](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:02:53 dist-test-slave-nmsz krb5kdc[8832](info): set up 2 sockets
Sep 29 19:02:53 dist-test-slave-nmsz krb5kdc[8832](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 19:02:54.819006 8825 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Sep 29 19:02:54 dist-test-slave-nmsz krb5kdc[8832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:02:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:02:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.158.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
I20260929 19:02:54.931123 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:32861
--webserver_interface=127.8.158.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:34197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:32861
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:02:55.007953 8845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:55.008141 8845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:55.008163 8845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:55.009203 8845 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:02:55.009238 8845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:55.009253 8845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:55.009265 8845 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:02:55.009277 8845 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:02:55.010648 8845 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:34197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:32861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:32861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.158.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:55.010825 8845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:55.010977 8845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:55.016077 8851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:55.016182 8845 server_base.cc:1034] running on GCE node
W20260929 19:02:55.016142 8850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:55.016115 8853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:55.016526 8845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:55.016808 8845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:55.017972 8845 hybrid_clock.cc:648] HybridClock initialized: now 1790708575017954 us; error 39 us; skew 500 ppm
Sep 29 19:02:55 dist-test-slave-nmsz krb5kdc[8832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708575, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:55.020442 8845 init.cc:377] Logged in from keytab as kudu/127.8.158.126@KRBTEST.COM (short username kudu)
I20260929 19:02:55.020882 8845 webserver.cc:460] Webserver started at http://127.8.158.126:44441/ using document root <none> and password file <none>
I20260929 19:02:55.021109 8845 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:55.021175 8845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:55.021323 8845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:55.022305 8845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "b2244aa7332f416588d2f441bba08b43"
format_stamp: "Formatted at 2026-09-29 19:02:55 on dist-test-slave-nmsz"
server_key: "2bb992f14916b4f8ae9f37c9d108bfc1"
server_key_iv: "e42f7859cbc89909e2a8c2dfd2ee9bfb"
server_key_version: "encryptionkey@0"
I20260929 19:02:55.022727 8845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "b2244aa7332f416588d2f441bba08b43"
format_stamp: "Formatted at 2026-09-29 19:02:55 on dist-test-slave-nmsz"
server_key: "2bb992f14916b4f8ae9f37c9d108bfc1"
server_key_iv: "e42f7859cbc89909e2a8c2dfd2ee9bfb"
server_key_version: "encryptionkey@0"
I20260929 19:02:55.024099 8845 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 19:02:55.024803 8860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:55.025002 8845 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:02:55.025098 8845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "b2244aa7332f416588d2f441bba08b43"
format_stamp: "Formatted at 2026-09-29 19:02:55 on dist-test-slave-nmsz"
server_key: "2bb992f14916b4f8ae9f37c9d108bfc1"
server_key_iv: "e42f7859cbc89909e2a8c2dfd2ee9bfb"
server_key_version: "encryptionkey@0"
I20260929 19:02:55.025183 8845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:55.048918 8845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:55.051234 8845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:55.051400 8845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:55.055485 8845 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.126:32861
I20260929 19:02:55.055524 8912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.126:32861 every 8 connection(s)
I20260929 19:02:55.055850 8845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:02:55.056650 8913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:55.058813 8913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2244aa7332f416588d2f441bba08b43: Bootstrap starting.
I20260929 19:02:55.059304 8913 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2244aa7332f416588d2f441bba08b43: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:55.059540 8913 log.cc:824] T 00000000000000000000000000000000 P b2244aa7332f416588d2f441bba08b43: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:55.060160 8913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2244aa7332f416588d2f441bba08b43: No bootstrap required, opened a new log
I20260929 19:02:55.061626 8913 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2244aa7332f416588d2f441bba08b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2244aa7332f416588d2f441bba08b43" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 32861 } }
I20260929 19:02:55.061762 8913 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2244aa7332f416588d2f441bba08b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:55.061784 8913 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2244aa7332f416588d2f441bba08b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2244aa7332f416588d2f441bba08b43, State: Initialized, Role: FOLLOWER
I20260929 19:02:55.061916 8913 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2244aa7332f416588d2f441bba08b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2244aa7332f416588d2f441bba08b43" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 32861 } }
I20260929 19:02:55.061996 8913 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2244aa7332f416588d2f441bba08b43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:02:55.062019 8913 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2244aa7332f416588d2f441bba08b43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:02:55.062047 8913 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2244aa7332f416588d2f441bba08b43 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:55.062572 8913 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2244aa7332f416588d2f441bba08b43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2244aa7332f416588d2f441bba08b43" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 32861 } }
I20260929 19:02:55.062656 8913 leader_election.cc:302] T 00000000000000000000000000000000 P b2244aa7332f416588d2f441bba08b43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2244aa7332f416588d2f441bba08b43; no voters:
I20260929 19:02:55.062772 8913 leader_election.cc:288] T 00000000000000000000000000000000 P b2244aa7332f416588d2f441bba08b43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:02:55.062837 8916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2244aa7332f416588d2f441bba08b43 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:02:55.063020 8916 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2244aa7332f416588d2f441bba08b43 [term 1 LEADER]: Becoming Leader. State: Replica: b2244aa7332f416588d2f441bba08b43, State: Running, Role: LEADER
I20260929 19:02:55.063032 8913 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2244aa7332f416588d2f441bba08b43 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:02:55.063119 8916 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2244aa7332f416588d2f441bba08b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2244aa7332f416588d2f441bba08b43" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 32861 } }
I20260929 19:02:55.063691 8918 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2244aa7332f416588d2f441bba08b43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2244aa7332f416588d2f441bba08b43. Latest consensus state: current_term: 1 leader_uuid: "b2244aa7332f416588d2f441bba08b43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2244aa7332f416588d2f441bba08b43" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 32861 } } }
I20260929 19:02:55.063786 8918 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2244aa7332f416588d2f441bba08b43 [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:55.063946 8917 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2244aa7332f416588d2f441bba08b43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2244aa7332f416588d2f441bba08b43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2244aa7332f416588d2f441bba08b43" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 32861 } } }
I20260929 19:02:55.064021 8917 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2244aa7332f416588d2f441bba08b43 [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:55.064024 8925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:02:55.064523 8925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:02:55.065443 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 8845
I20260929 19:02:55.065544 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:02:55.066294 8825 external_mini_cluster.cc:1428] Setting key 0193b8db633c9ed284b51de3fb2295eb
I20260929 19:02:55.067549 8925 catalog_manager.cc:1347] Generated new cluster ID: 0399065a94be4f6c9f88abbb441352d8
I20260929 19:02:55.067605 8925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 19:02:55 dist-test-slave-nmsz krb5kdc[8832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:02:55.079030 8925 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:02:55.079639 8925 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:02:55.083359 8935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:55.067138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59776 (local address 127.8.158.126:32861)
0929 19:02:55.067300 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:55.067309 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:55.067632 (+ 323us) server_negotiation.cc:407] Connection header received
0929 19:02:55.069702 (+ 2070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:55.069709 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:55.069762 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:55.069920 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:55.071758 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:55.073209 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:55.074603 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:55.075201 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:55.075310 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:55.079495 (+ 4185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:55.079498 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:55.079499 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:55.079509 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:55.081424 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:55.081909 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:55.081912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:55.081912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:55.082020 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:55.082296 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:55.082296 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:55.082297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:55.082557 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:55.082726 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:55.082992 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 19:02:55.083123 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":78,"threads_started":1}
I20260929 19:02:55.085098 8925 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2244aa7332f416588d2f441bba08b43: Generated new TSK 0
I20260929 19:02:55.085247 8925 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.158.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab.
Entry for principal kudu/127.8.158.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab.
I20260929 19:02:55.112936 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.65:0
--local_ip_for_outbound_sockets=127.8.158.65
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:32861
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab
--principal=kudu/127.8.158.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:34197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:02:55.191720 8939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:55.191955 8939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:55.191982 8939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:55.193637 8939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:55.193693 8939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:55.193730 8939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.65
I20260929 19:02:55.195348 8939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:34197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab
--principal=kudu/127.8.158.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:32861
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.65
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:55.195533 8939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:55.195694 8939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:55.200508 8945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:55.200503 8947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:55.200532 8944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:55.200698 8939 server_base.cc:1034] running on GCE node
I20260929 19:02:55.200911 8939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:55.201152 8939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:55.202347 8939 hybrid_clock.cc:648] HybridClock initialized: now 1790708575202307 us; error 42 us; skew 500 ppm
Sep 29 19:02:55 dist-test-slave-nmsz krb5kdc[8832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708575, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:55.204825 8939 init.cc:377] Logged in from keytab as kudu/127.8.158.65@KRBTEST.COM (short username kudu)
I20260929 19:02:55.205173 8939 webserver.cc:460] Webserver started at http://127.8.158.65:45629/ using document root <none> and password file <none>
I20260929 19:02:55.205391 8939 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:55.205436 8939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:55.205569 8939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:55.206622 8939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-0/data/instance:
uuid: "5dc9f34c4e9948a1acfa7e0380355626"
format_stamp: "Formatted at 2026-09-29 19:02:55 on dist-test-slave-nmsz"
server_key: "add776380da90466dc5cac0e826e1ce9"
server_key_iv: "a45fbd907377199d028efbd01ab6e618"
server_key_version: "encryptionkey@0"
I20260929 19:02:55.206969 8939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance:
uuid: "5dc9f34c4e9948a1acfa7e0380355626"
format_stamp: "Formatted at 2026-09-29 19:02:55 on dist-test-slave-nmsz"
server_key: "add776380da90466dc5cac0e826e1ce9"
server_key_iv: "a45fbd907377199d028efbd01ab6e618"
server_key_version: "encryptionkey@0"
I20260929 19:02:55.208235 8939 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:02:55.208966 8954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:55.209167 8939 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:02:55.209233 8939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-0/wal
uuid: "5dc9f34c4e9948a1acfa7e0380355626"
format_stamp: "Formatted at 2026-09-29 19:02:55 on dist-test-slave-nmsz"
server_key: "add776380da90466dc5cac0e826e1ce9"
server_key_iv: "a45fbd907377199d028efbd01ab6e618"
server_key_version: "encryptionkey@0"
I20260929 19:02:55.209317 8939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:55.221019 8939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:55.223239 8939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:55.223343 8939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:55.223510 8939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:02:55.223798 8939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:02:55.223832 8939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:55.223852 8939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:02:55.223901 8939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:55.229573 8939 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.65:36193
I20260929 19:02:55.229636 9067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.65:36193 every 8 connection(s)
I20260929 19:02:55.229910 8939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
Sep 29 19:02:55 dist-test-slave-nmsz krb5kdc[8832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708575, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:02:55.237461 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 8939
I20260929 19:02:55.237588 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance
I20260929 19:02:55.237916 8825 external_mini_cluster.cc:1428] Setting key 87fd5c1227832e4cf6768624a84436c3
I20260929 19:02:55.239440 8935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:55.230616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:44621 (local address 127.8.158.126:32861)
0929 19:02:55.230676 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:55.230678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:55.230865 (+ 187us) server_negotiation.cc:407] Connection header received
0929 19:02:55.231385 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:55.231387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:55.231399 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:55.231447 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:55.232371 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:55.232883 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:55.233835 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:55.234052 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:55.234134 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:55.236096 (+ 1962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:55.236097 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:55.236097 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:55.236103 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:55.237892 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:55.238413 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:55.238415 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:55.238415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:55.238483 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:55.238750 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:55.238751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:55.238751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:55.238892 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:55.239005 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:55.239262 (+ 257us) server_negotiation.cc:299] Negotiation successful
0929 19:02:55.239354 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 19:02:55.239645 9070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:55.230602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:32861 (local address 127.8.158.65:44621)
0929 19:02:55.230744 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:55.230776 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:55.231251 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:55.231527 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:55.231530 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:55.232073 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:55.232261 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:55.232265 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:55.232972 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:55.232973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:55.233731 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:55.233733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:55.234142 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:55.234144 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:55.234259 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:55.234736 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:02:55.234743 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:55.236013 (+ 1270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:55.237992 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:55.237998 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:55.237999 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:55.238351 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:55.238552 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:55.238553 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:55.238553 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:55.238652 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:55.238999 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:55.239005 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:55.239134 (+ 129us) client_negotiation.cc:769] Sending connection context
0929 19:02:55.239242 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 19:02:55.239387 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
I20260929 19:02:55.240409 9068 heartbeater.cc:347] Connected to a master server at 127.8.158.126:32861
I20260929 19:02:55.240525 9068 heartbeater.cc:464] Registering TS with master...
I20260929 19:02:55.240686 9068 heartbeater.cc:511] Master 127.8.158.126:32861 requested a full tablet report, sending...
I20260929 19:02:55.241250 8877 ts_manager.cc:194] Registered new tserver with Master: 5dc9f34c4e9948a1acfa7e0380355626 (127.8.158.65:36193)
I20260929 19:02:55.241917 8877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.65@KRBTEST.COM'} at 127.8.158.65:44621
WARNING: no policy specified for kudu/127.8.158.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab.
Entry for principal kudu/127.8.158.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab.
I20260929 19:02:55.263299 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.66:0
--local_ip_for_outbound_sockets=127.8.158.66
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:32861
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab
--principal=kudu/127.8.158.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:34197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:02:55.342852 9073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:55.343070 9073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:55.343106 9073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:55.344240 9073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:55.344306 9073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:55.344352 9073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.66
I20260929 19:02:55.345809 9073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:34197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab
--principal=kudu/127.8.158.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:32861
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.66
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:55.346071 9073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:55.346295 9073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:55.351095 9079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:55.351097 9081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:55.351097 9078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:55.351429 9073 server_base.cc:1034] running on GCE node
I20260929 19:02:55.351641 9073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:55.351888 9073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:55.353058 9073 hybrid_clock.cc:648] HybridClock initialized: now 1790708575353032 us; error 36 us; skew 500 ppm
Sep 29 19:02:55 dist-test-slave-nmsz krb5kdc[8832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708575, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:55.355697 9073 init.cc:377] Logged in from keytab as kudu/127.8.158.66@KRBTEST.COM (short username kudu)
I20260929 19:02:55.356104 9073 webserver.cc:460] Webserver started at http://127.8.158.66:36687/ using document root <none> and password file <none>
I20260929 19:02:55.356352 9073 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:55.356422 9073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:55.356564 9073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:55.357576 9073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-1/data/instance:
uuid: "b79c53a96be942c6aaaf8b98dffa3cbb"
format_stamp: "Formatted at 2026-09-29 19:02:55 on dist-test-slave-nmsz"
server_key: "96d40401a9cc660b0e202f624d875c42"
server_key_iv: "4d09de4f345dd101b96a12e11090f66b"
server_key_version: "encryptionkey@0"
I20260929 19:02:55.357936 9073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance:
uuid: "b79c53a96be942c6aaaf8b98dffa3cbb"
format_stamp: "Formatted at 2026-09-29 19:02:55 on dist-test-slave-nmsz"
server_key: "96d40401a9cc660b0e202f624d875c42"
server_key_iv: "4d09de4f345dd101b96a12e11090f66b"
server_key_version: "encryptionkey@0"
I20260929 19:02:55.359341 9073 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:02:55.360155 9088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:55.360334 9073 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:02:55.360443 9073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-1/wal
uuid: "b79c53a96be942c6aaaf8b98dffa3cbb"
format_stamp: "Formatted at 2026-09-29 19:02:55 on dist-test-slave-nmsz"
server_key: "96d40401a9cc660b0e202f624d875c42"
server_key_iv: "4d09de4f345dd101b96a12e11090f66b"
server_key_version: "encryptionkey@0"
I20260929 19:02:55.360533 9073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:55.378136 9073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:55.380214 9073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:55.380465 9073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:55.380734 9073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:02:55.381057 9073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:02:55.381114 9073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:55.381150 9073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:02:55.381179 9073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:55.387135 9073 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.66:46799
I20260929 19:02:55.387188 9201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.66:46799 every 8 connection(s)
I20260929 19:02:55.387495 9073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
I20260929 19:02:55.387775 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 9073
I20260929 19:02:55.387887 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance
I20260929 19:02:55.388087 8825 external_mini_cluster.cc:1428] Setting key bcfe2e2b83e64c21240a054867ad7668
Sep 29 19:02:55 dist-test-slave-nmsz krb5kdc[8832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708575, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.66@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.158.67@KRBTEST.COM; defaulting to no policy
I20260929 19:02:55.397698 8935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:55.388252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:50327 (local address 127.8.158.126:32861)
0929 19:02:55.388305 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:55.388308 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:55.388576 (+ 268us) server_negotiation.cc:407] Connection header received
0929 19:02:55.389226 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:55.389227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:55.389241 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:55.389327 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:55.390623 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:55.391097 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:55.392191 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:55.392443 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:55.392555 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:55.394920 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:55.394921 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:55.394921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:55.394928 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:55.396439 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:55.396789 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:55.396790 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:55.396790 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:55.396864 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:55.397148 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:55.397149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:55.397149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:55.397260 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:55.397317 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:55.397569 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 19:02:55.397631 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 19:02:55.397928 9204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:55.388254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:32861 (local address 127.8.158.66:50327)
0929 19:02:55.388451 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:55.388500 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:55.389092 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:55.389430 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:55.389433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:55.390164 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:55.390508 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:55.390515 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:55.391192 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:55.391193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:55.392090 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:55.392091 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:55.392557 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:55.392558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:55.392700 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:55.393233 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:02:55.393241 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:55.394805 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:55.396516 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:55.396521 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:55.396523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:55.396721 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:55.396933 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:55.396933 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:55.396933 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:55.397098 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:55.397325 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:55.397337 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:55.397452 (+ 115us) client_negotiation.cc:769] Sending connection context
0929 19:02:55.397539 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 19:02:55.397651 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":90,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.67@KRBTEST.COM" created.
I20260929 19:02:55.398483 9202 heartbeater.cc:347] Connected to a master server at 127.8.158.126:32861
I20260929 19:02:55.398577 9202 heartbeater.cc:464] Registering TS with master...
I20260929 19:02:55.398775 9202 heartbeater.cc:511] Master 127.8.158.126:32861 requested a full tablet report, sending...
I20260929 19:02:55.399217 8877 ts_manager.cc:194] Registered new tserver with Master: b79c53a96be942c6aaaf8b98dffa3cbb (127.8.158.66:46799)
I20260929 19:02:55.399643 8877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.66@KRBTEST.COM'} at 127.8.158.66:50327
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab.
Entry for principal kudu/127.8.158.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab.
I20260929 19:02:55.413180 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.67:0
--local_ip_for_outbound_sockets=127.8.158.67
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:32861
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab
--principal=kudu/127.8.158.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:34197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:02:55.492718 9207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:55.492942 9207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:55.492980 9207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:55.494091 9207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:55.494156 9207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:55.494200 9207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.67
I20260929 19:02:55.495631 9207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:34197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab
--principal=kudu/127.8.158.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:32861
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.67
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:55.495885 9207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:55.496109 9207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:55.500953 9212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:55.500988 9215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:55.501391 9213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:55.501400 9207 server_base.cc:1034] running on GCE node
I20260929 19:02:55.501672 9207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:55.501907 9207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:55.503067 9207 hybrid_clock.cc:648] HybridClock initialized: now 1790708575503059 us; error 38 us; skew 500 ppm
Sep 29 19:02:55 dist-test-slave-nmsz krb5kdc[8832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708575, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:55.505650 9207 init.cc:377] Logged in from keytab as kudu/127.8.158.67@KRBTEST.COM (short username kudu)
I20260929 19:02:55.506089 9207 webserver.cc:460] Webserver started at http://127.8.158.67:35397/ using document root <none> and password file <none>
I20260929 19:02:55.506357 9207 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:55.506438 9207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:55.506565 9207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:55.507601 9207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-2/data/instance:
uuid: "e54f7ea442864802b143d24ddd2cdf87"
format_stamp: "Formatted at 2026-09-29 19:02:55 on dist-test-slave-nmsz"
server_key: "71f34917a7513fe8d546412d55cbbad6"
server_key_iv: "590b109580776c6bc378e8dc8fe01dba"
server_key_version: "encryptionkey@0"
I20260929 19:02:55.507957 9207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance:
uuid: "e54f7ea442864802b143d24ddd2cdf87"
format_stamp: "Formatted at 2026-09-29 19:02:55 on dist-test-slave-nmsz"
server_key: "71f34917a7513fe8d546412d55cbbad6"
server_key_iv: "590b109580776c6bc378e8dc8fe01dba"
server_key_version: "encryptionkey@0"
I20260929 19:02:55.509330 9207 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:02:55.510113 9222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:55.510363 9207 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:02:55.510468 9207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-2/wal
uuid: "e54f7ea442864802b143d24ddd2cdf87"
format_stamp: "Formatted at 2026-09-29 19:02:55 on dist-test-slave-nmsz"
server_key: "71f34917a7513fe8d546412d55cbbad6"
server_key_iv: "590b109580776c6bc378e8dc8fe01dba"
server_key_version: "encryptionkey@0"
I20260929 19:02:55.510548 9207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:55.519233 9207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:55.520962 9207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:55.521102 9207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:55.521294 9207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:02:55.521569 9207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:02:55.521620 9207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:55.521657 9207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:02:55.521687 9207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:55.527566 9207 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.67:35579
I20260929 19:02:55.527609 9335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.67:35579 every 8 connection(s)
I20260929 19:02:55.527911 9207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
Sep 29 19:02:55 dist-test-slave-nmsz krb5kdc[8832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708575, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.67@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:02:55.536803 8935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:55.528617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:57805 (local address 127.8.158.126:32861)
0929 19:02:55.528672 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:55.528674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:55.528878 (+ 204us) server_negotiation.cc:407] Connection header received
0929 19:02:55.529301 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:55.529303 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:55.529313 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:55.529378 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:55.530246 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:55.530837 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:55.531866 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:55.532044 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:55.532096 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:55.534167 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:55.534169 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:55.534170 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:55.534177 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:55.535444 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:55.535852 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:55.535853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:55.535854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:55.535924 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:55.536171 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:55.536172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:55.536172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:55.536293 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:55.536368 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:55.536638 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 19:02:55.536712 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260929 19:02:55.537003 9338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:55.528583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:32861 (local address 127.8.158.67:57805)
0929 19:02:55.528724 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:55.528769 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:55.529224 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:55.529429 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:55.529431 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:55.529966 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:55.530141 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:55.530145 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:55.530936 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:55.530937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:55.531787 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:55.531789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:55.532117 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:55.532118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:55.532218 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:55.532659 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:02:55.532668 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:55.534035 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:55.535531 (+ 1496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:55.535537 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:55.535540 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:55.535755 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:55.535992 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:55.535992 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:55.535993 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:55.536102 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:55.536361 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:55.536366 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:55.536511 (+ 145us) client_negotiation.cc:769] Sending connection context
0929 19:02:55.536617 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 19:02:55.536765 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":77,"threads_started":1}
I20260929 19:02:55.537501 9336 heartbeater.cc:347] Connected to a master server at 127.8.158.126:32861
I20260929 19:02:55.537501 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 9207
I20260929 19:02:55.537582 9336 heartbeater.cc:464] Registering TS with master...
I20260929 19:02:55.537609 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance
I20260929 19:02:55.537760 9336 heartbeater.cc:511] Master 127.8.158.126:32861 requested a full tablet report, sending...
I20260929 19:02:55.537771 8825 external_mini_cluster.cc:1428] Setting key 5bd9633d8d7b15c2ff6c6b077fe190fc
I20260929 19:02:55.538180 8877 ts_manager.cc:194] Registered new tserver with Master: e54f7ea442864802b143d24ddd2cdf87 (127.8.158.67:35579)
I20260929 19:02:55.538672 8877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.67@KRBTEST.COM'} at 127.8.158.67:57805
I20260929 19:02:55.538913 8825 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 19:02:55 dist-test-slave-nmsz krb5kdc[8832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708575, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 19:02:55 dist-test-slave-nmsz krb5kdc[8832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708575, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.158.65@KRBTEST.COM
I20260929 19:02:55.561057 9343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:55.553150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46888 (local address 127.8.158.65:36193)
0929 19:02:55.553324 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:55.553329 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:55.553341 (+ 12us) server_negotiation.cc:407] Connection header received
0929 19:02:55.553372 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:55.553374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:55.553389 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:55.553486 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:55.554101 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:55.554960 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:55.555780 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:55.556137 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:55.556228 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:55.558585 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:55.558591 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:55.558592 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:55.558599 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:55.560050 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:55.560376 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:55.560378 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:55.560379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:55.560435 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:55.560605 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:55.560606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:55.560606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:55.560727 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:55.560831 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:55.560917 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 19:02:55.560978 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":77,"threads_started":1}
W20260929 19:02:55.561471 8982 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46888
Sep 29 19:02:55 dist-test-slave-nmsz krb5kdc[8832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 19:02:55 dist-test-slave-nmsz krb5kdc[8832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.158.65@KRBTEST.COM
I20260929 19:02:55.583494 9343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:55.576428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46894 (local address 127.8.158.65:36193)
0929 19:02:55.576510 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:55.576512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:55.576522 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:02:55.576581 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:55.576582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:55.576599 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:55.576663 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:55.577205 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:55.578037 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:55.578883 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:55.579101 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:55.579182 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:55.581295 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:55.581296 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:55.581296 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:55.581302 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:55.582556 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:55.582853 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:55.582854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:55.582854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:55.582908 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:55.583064 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:55.583065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:55.583065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:55.583143 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:55.583225 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:55.583315 (+ 90us) server_negotiation.cc:299] Negotiation successful
0929 19:02:55.583400 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 19:02:55.583992 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 8939
I20260929 19:02:55.588819 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 9073
I20260929 19:02:55.593314 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 9207
I20260929 19:02:55.597306 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 8845
2026-09-29T19:02:55Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1868 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:02:55 dist-test-slave-nmsz krb5kdc[9350](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:02:55 dist-test-slave-nmsz krb5kdc[9350](info): set up 2 sockets
Sep 29 19:02:55 dist-test-slave-nmsz krb5kdc[9350](info): commencing operation
krb5kdc: starting...
W20260929 19:02:56.637900 8825 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Sep 29 19:02:56 dist-test-slave-nmsz krb5kdc[9350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:02:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:02:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.158.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
I20260929 19:02:56.749854 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:33913
--webserver_interface=127.8.158.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:33785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:33913
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:02:56.827739 9363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:56.827950 9363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:56.827983 9363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:56.829084 9363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:02:56.829150 9363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:56.829173 9363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:56.829195 9363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:02:56.829216 9363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:02:56.830757 9363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:33785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:33913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:33913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.158.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:56.831012 9363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:56.831230 9363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:56.835745 9371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:56.835752 9369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:56.835799 9368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:56.835959 9363 server_base.cc:1034] running on GCE node
I20260929 19:02:56.836216 9363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:56.836516 9363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:56.837684 9363 hybrid_clock.cc:648] HybridClock initialized: now 1790708576837661 us; error 34 us; skew 500 ppm
Sep 29 19:02:56 dist-test-slave-nmsz krb5kdc[9350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708576, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:56.840093 9363 init.cc:377] Logged in from keytab as kudu/127.8.158.126@KRBTEST.COM (short username kudu)
I20260929 19:02:56.840579 9363 webserver.cc:460] Webserver started at http://127.8.158.126:32859/ using document root <none> and password file <none>
I20260929 19:02:56.840821 9363 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:56.840889 9363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:56.841037 9363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:56.842041 9363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "185fb590bb494e51b439bc3505b02be2"
format_stamp: "Formatted at 2026-09-29 19:02:56 on dist-test-slave-nmsz"
server_key: "19081aa25fa0e4507b3c0533f9e75df1"
server_key_iv: "1e41863691d7c1137ebd78db08f9c8c0"
server_key_version: "encryptionkey@0"
I20260929 19:02:56.842478 9363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "185fb590bb494e51b439bc3505b02be2"
format_stamp: "Formatted at 2026-09-29 19:02:56 on dist-test-slave-nmsz"
server_key: "19081aa25fa0e4507b3c0533f9e75df1"
server_key_iv: "1e41863691d7c1137ebd78db08f9c8c0"
server_key_version: "encryptionkey@0"
I20260929 19:02:56.843732 9363 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:02:56.844434 9378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:56.844605 9363 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:02:56.844702 9363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "185fb590bb494e51b439bc3505b02be2"
format_stamp: "Formatted at 2026-09-29 19:02:56 on dist-test-slave-nmsz"
server_key: "19081aa25fa0e4507b3c0533f9e75df1"
server_key_iv: "1e41863691d7c1137ebd78db08f9c8c0"
server_key_version: "encryptionkey@0"
I20260929 19:02:56.844796 9363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:56.855665 9363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:56.857556 9363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:56.857714 9363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:56.861706 9363 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.126:33913
I20260929 19:02:56.861752 9430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.126:33913 every 8 connection(s)
I20260929 19:02:56.862087 9363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:02:56.862962 9431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:56.863773 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 9363
I20260929 19:02:56.863888 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:02:56.864044 8825 external_mini_cluster.cc:1428] Setting key 33223088758ace7a51162f19d3cd77db
I20260929 19:02:56.866109 9431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 185fb590bb494e51b439bc3505b02be2: Bootstrap starting.
I20260929 19:02:56.866756 9431 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 185fb590bb494e51b439bc3505b02be2: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:56.867043 9431 log.cc:824] T 00000000000000000000000000000000 P 185fb590bb494e51b439bc3505b02be2: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:56.867689 9431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 185fb590bb494e51b439bc3505b02be2: No bootstrap required, opened a new log
I20260929 19:02:56.869585 9431 raft_consensus.cc:348] T 00000000000000000000000000000000 P 185fb590bb494e51b439bc3505b02be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "185fb590bb494e51b439bc3505b02be2" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 33913 } }
I20260929 19:02:56.869738 9431 raft_consensus.cc:374] T 00000000000000000000000000000000 P 185fb590bb494e51b439bc3505b02be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:56.869805 9431 raft_consensus.cc:729] T 00000000000000000000000000000000 P 185fb590bb494e51b439bc3505b02be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 185fb590bb494e51b439bc3505b02be2, State: Initialized, Role: FOLLOWER
I20260929 19:02:56.869956 9431 consensus_queue.cc:260] T 00000000000000000000000000000000 P 185fb590bb494e51b439bc3505b02be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "185fb590bb494e51b439bc3505b02be2" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 33913 } }
I20260929 19:02:56.870050 9431 raft_consensus.cc:388] T 00000000000000000000000000000000 P 185fb590bb494e51b439bc3505b02be2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:02:56.870083 9431 raft_consensus.cc:482] T 00000000000000000000000000000000 P 185fb590bb494e51b439bc3505b02be2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:02:56.870122 9431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 185fb590bb494e51b439bc3505b02be2 [term 0 FOLLOWER]: Advancing to term 1
Sep 29 19:02:56 dist-test-slave-nmsz krb5kdc[9350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:02:56.870836 9431 raft_consensus.cc:504] T 00000000000000000000000000000000 P 185fb590bb494e51b439bc3505b02be2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "185fb590bb494e51b439bc3505b02be2" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 33913 } }
I20260929 19:02:56.870950 9431 leader_election.cc:302] T 00000000000000000000000000000000 P 185fb590bb494e51b439bc3505b02be2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 185fb590bb494e51b439bc3505b02be2; no voters:
I20260929 19:02:56.871119 9431 leader_election.cc:288] T 00000000000000000000000000000000 P 185fb590bb494e51b439bc3505b02be2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:02:56.871177 9436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 185fb590bb494e51b439bc3505b02be2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:02:56.871325 9436 raft_consensus.cc:686] T 00000000000000000000000000000000 P 185fb590bb494e51b439bc3505b02be2 [term 1 LEADER]: Becoming Leader. State: Replica: 185fb590bb494e51b439bc3505b02be2, State: Running, Role: LEADER
I20260929 19:02:56.871417 9436 consensus_queue.cc:237] T 00000000000000000000000000000000 P 185fb590bb494e51b439bc3505b02be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "185fb590bb494e51b439bc3505b02be2" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 33913 } }
I20260929 19:02:56.871436 9431 sys_catalog.cc:564] T 00000000000000000000000000000000 P 185fb590bb494e51b439bc3505b02be2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:02:56.871862 9436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 185fb590bb494e51b439bc3505b02be2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 185fb590bb494e51b439bc3505b02be2. Latest consensus state: current_term: 1 leader_uuid: "185fb590bb494e51b439bc3505b02be2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "185fb590bb494e51b439bc3505b02be2" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 33913 } } }
I20260929 19:02:56.871965 9436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 185fb590bb494e51b439bc3505b02be2 [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:56.871876 9437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 185fb590bb494e51b439bc3505b02be2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "185fb590bb494e51b439bc3505b02be2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "185fb590bb494e51b439bc3505b02be2" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 33913 } } }
I20260929 19:02:56.872110 9437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 185fb590bb494e51b439bc3505b02be2 [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:56.872162 9441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:02:56.873029 9441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:02:56.874758 9441 catalog_manager.cc:1347] Generated new cluster ID: 9906e4604c6848898820f0d42b91bf69
I20260929 19:02:56.874811 9441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:02:56.875092 9434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:56.864535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60088 (local address 127.8.158.126:33913)
0929 19:02:56.864699 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:56.864706 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:56.864730 (+ 24us) server_negotiation.cc:407] Connection header received
0929 19:02:56.865122 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:56.865128 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:56.865170 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:56.865276 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:56.866092 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:56.867198 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:56.868023 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:56.868431 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:56.868551 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:56.871131 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:56.871135 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:56.871136 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:56.871148 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:56.873400 (+ 2252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:56.873840 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:56.873843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:56.873843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:56.873937 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:56.874144 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:56.874145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:56.874145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:56.874494 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:56.874650 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:56.874760 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 19:02:56.874892 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":79,"threads_started":1}
I20260929 19:02:56.903213 9441 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:02:56.903759 9441 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:02:56.909883 9441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 185fb590bb494e51b439bc3505b02be2: Generated new TSK 0
I20260929 19:02:56.910070 9441 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.158.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab.
Entry for principal kudu/127.8.158.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab.
I20260929 19:02:56.937709 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.65:0
--local_ip_for_outbound_sockets=127.8.158.65
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:33913
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab
--principal=kudu/127.8.158.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:33785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:02:57.031167 9457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:57.031378 9457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:57.031401 9457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:57.032694 9457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:57.032735 9457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:57.032763 9457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.65
I20260929 19:02:57.034287 9457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:33785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab
--principal=kudu/127.8.158.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:33913
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.65
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:57.034576 9457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:57.034816 9457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:57.039547 9462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:57.039600 9463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:57.039795 9457 server_base.cc:1034] running on GCE node
W20260929 19:02:57.039535 9465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:57.040058 9457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:57.040294 9457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:57.041452 9457 hybrid_clock.cc:648] HybridClock initialized: now 1790708577041462 us; error 49 us; skew 500 ppm
Sep 29 19:02:57 dist-test-slave-nmsz krb5kdc[9350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708577, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:57.043934 9457 init.cc:377] Logged in from keytab as kudu/127.8.158.65@KRBTEST.COM (short username kudu)
I20260929 19:02:57.044297 9457 webserver.cc:460] Webserver started at http://127.8.158.65:44305/ using document root <none> and password file <none>
I20260929 19:02:57.044480 9457 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:57.044519 9457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:57.044600 9457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:57.045516 9457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-0/data/instance:
uuid: "ba4f93b763594b82b9884e440a60aad0"
format_stamp: "Formatted at 2026-09-29 19:02:57 on dist-test-slave-nmsz"
server_key: "9f618fc2b68a81dc87e85719acc97899"
server_key_iv: "7e51ff1f834a706a2bd8bff5456b09ec"
server_key_version: "encryptionkey@0"
I20260929 19:02:57.045804 9457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance:
uuid: "ba4f93b763594b82b9884e440a60aad0"
format_stamp: "Formatted at 2026-09-29 19:02:57 on dist-test-slave-nmsz"
server_key: "9f618fc2b68a81dc87e85719acc97899"
server_key_iv: "7e51ff1f834a706a2bd8bff5456b09ec"
server_key_version: "encryptionkey@0"
I20260929 19:02:57.047080 9457 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 19:02:57.047816 9472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:57.047964 9457 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:02:57.048043 9457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-0/wal
uuid: "ba4f93b763594b82b9884e440a60aad0"
format_stamp: "Formatted at 2026-09-29 19:02:57 on dist-test-slave-nmsz"
server_key: "9f618fc2b68a81dc87e85719acc97899"
server_key_iv: "7e51ff1f834a706a2bd8bff5456b09ec"
server_key_version: "encryptionkey@0"
I20260929 19:02:57.048095 9457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:57.069718 9457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:57.071588 9457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:57.071693 9457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:57.071936 9457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:02:57.072329 9457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:02:57.072367 9457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:57.072413 9457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:02:57.072439 9457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:57.077876 9457 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.65:42511
I20260929 19:02:57.077937 9585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.65:42511 every 8 connection(s)
I20260929 19:02:57.078241 9457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
I20260929 19:02:57.082506 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 9457
I20260929 19:02:57.082602 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance
I20260929 19:02:57.082777 8825 external_mini_cluster.cc:1428] Setting key b54ba5e89ca0abf6adc27d3386e352b3
Sep 29 19:02:57 dist-test-slave-nmsz krb5kdc[9350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708577, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:02:57.087543 9434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:57.078960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:58029 (local address 127.8.158.126:33913)
0929 19:02:57.079034 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:57.079036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:57.079193 (+ 157us) server_negotiation.cc:407] Connection header received
0929 19:02:57.080047 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:57.080048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:57.080058 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:57.080106 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:57.080654 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:57.081128 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:57.082069 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:57.082253 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:57.082301 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:57.084694 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:57.084696 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:57.084696 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:57.084703 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:57.086137 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:57.086620 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:57.086622 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:57.086622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:57.086678 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:57.086930 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:57.086931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:57.086931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:57.087041 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:57.087117 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:57.087369 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 19:02:57.087451 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 19:02:57.087731 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:57.078921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:33913 (local address 127.8.158.65:58029)
0929 19:02:57.079069 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:57.079101 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:57.079958 (+ 857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:57.080163 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:57.080166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:57.080414 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:57.080585 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:57.080589 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:57.081205 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:57.081206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:57.081996 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:57.081998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:57.082372 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:57.082373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:57.082519 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:57.083092 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:02:57.083102 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:57.084598 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:57.086237 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:57.086244 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:57.086246 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:57.086533 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:57.086744 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:57.086745 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:57.086745 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:57.086855 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:57.087107 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:57.087112 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:57.087243 (+ 131us) client_negotiation.cc:769] Sending connection context
0929 19:02:57.087348 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 19:02:57.087501 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":90,"threads_started":1}
I20260929 19:02:57.088282 9586 heartbeater.cc:347] Connected to a master server at 127.8.158.126:33913
I20260929 19:02:57.088372 9586 heartbeater.cc:464] Registering TS with master...
I20260929 19:02:57.088541 9586 heartbeater.cc:511] Master 127.8.158.126:33913 requested a full tablet report, sending...
I20260929 19:02:57.089104 9395 ts_manager.cc:194] Registered new tserver with Master: ba4f93b763594b82b9884e440a60aad0 (127.8.158.65:42511)
I20260929 19:02:57.090039 9395 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.65@KRBTEST.COM'} at 127.8.158.65:58029
WARNING: no policy specified for kudu/127.8.158.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab.
Entry for principal kudu/127.8.158.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab.
I20260929 19:02:57.108841 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.66:0
--local_ip_for_outbound_sockets=127.8.158.66
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:33913
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab
--principal=kudu/127.8.158.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:33785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:02:57.188403 9591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:57.188593 9591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:57.188616 9591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:57.189757 9591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:57.189798 9591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:57.189826 9591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.66
I20260929 19:02:57.191291 9591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:33785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab
--principal=kudu/127.8.158.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:33913
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.66
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:57.191522 9591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:57.191738 9591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:57.196698 9596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:57.196709 9599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:57.196698 9597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:57.196941 9591 server_base.cc:1034] running on GCE node
I20260929 19:02:57.197180 9591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:57.197431 9591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:57.198623 9591 hybrid_clock.cc:648] HybridClock initialized: now 1790708577198603 us; error 27 us; skew 500 ppm
Sep 29 19:02:57 dist-test-slave-nmsz krb5kdc[9350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708577, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:57.202356 9591 init.cc:377] Logged in from keytab as kudu/127.8.158.66@KRBTEST.COM (short username kudu)
I20260929 19:02:57.202741 9591 webserver.cc:460] Webserver started at http://127.8.158.66:39669/ using document root <none> and password file <none>
I20260929 19:02:57.202970 9591 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:57.203038 9591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:57.203169 9591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:57.204185 9591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-1/data/instance:
uuid: "e0968d9fe5fe4bc7ac85411767291d78"
format_stamp: "Formatted at 2026-09-29 19:02:57 on dist-test-slave-nmsz"
server_key: "bc46327b74e0e8bf6d5fb4c6930ff6b9"
server_key_iv: "a5309de7b405b4eb137746fbb70025a1"
server_key_version: "encryptionkey@0"
I20260929 19:02:57.204555 9591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance:
uuid: "e0968d9fe5fe4bc7ac85411767291d78"
format_stamp: "Formatted at 2026-09-29 19:02:57 on dist-test-slave-nmsz"
server_key: "bc46327b74e0e8bf6d5fb4c6930ff6b9"
server_key_iv: "a5309de7b405b4eb137746fbb70025a1"
server_key_version: "encryptionkey@0"
I20260929 19:02:57.205853 9591 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:02:57.206614 9606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:57.206776 9591 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:02:57.206873 9591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-1/wal
uuid: "e0968d9fe5fe4bc7ac85411767291d78"
format_stamp: "Formatted at 2026-09-29 19:02:57 on dist-test-slave-nmsz"
server_key: "bc46327b74e0e8bf6d5fb4c6930ff6b9"
server_key_iv: "a5309de7b405b4eb137746fbb70025a1"
server_key_version: "encryptionkey@0"
I20260929 19:02:57.206945 9591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:57.225008 9591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:57.227157 9591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:57.227336 9591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:57.227579 9591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:02:57.227890 9591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:02:57.227944 9591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:57.227985 9591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:02:57.228013 9591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:57.234078 9591 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.66:39215
I20260929 19:02:57.234118 9719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.66:39215 every 8 connection(s)
I20260929 19:02:57.234454 9591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
Sep 29 19:02:57 dist-test-slave-nmsz krb5kdc[9350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708577, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.66@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:02:57.243531 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 9591
I20260929 19:02:57.243651 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance
I20260929 19:02:57.243854 8825 external_mini_cluster.cc:1428] Setting key 966c18515ecac29547759eecb925dc93
I20260929 19:02:57.244526 9434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:57.235224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:59911 (local address 127.8.158.126:33913)
0929 19:02:57.235359 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:57.235362 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:57.235512 (+ 150us) server_negotiation.cc:407] Connection header received
0929 19:02:57.236590 (+ 1078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:57.236592 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:57.236604 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:57.236656 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:57.237247 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:57.237739 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:57.239012 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:57.239227 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:57.239324 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:57.241569 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:57.241571 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:57.241572 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:57.241580 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:57.242946 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:57.243399 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:57.243400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:57.243401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:57.243465 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:57.243786 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:57.243788 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:57.243788 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:57.243919 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:57.244002 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:57.244356 (+ 354us) server_negotiation.cc:299] Negotiation successful
0929 19:02:57.244438 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260929 19:02:57.244779 9722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:57.235148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:33913 (local address 127.8.158.66:59911)
0929 19:02:57.235387 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:57.235430 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:57.236478 (+ 1048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:57.236727 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:57.236731 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:57.236963 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:57.237143 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:57.237147 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:57.237865 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:57.237867 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:57.238895 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:57.238898 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:57.239324 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:57.239325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:57.239454 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:57.239918 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:02:57.239926 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:57.241464 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:57.243059 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:57.243065 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:57.243067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:57.243298 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:57.243552 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:57.243553 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:57.243553 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:57.243692 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:57.244028 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:57.244035 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:57.244198 (+ 163us) client_negotiation.cc:769] Sending connection context
0929 19:02:57.244313 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 19:02:57.244502 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":86,"threads_started":1}
I20260929 19:02:57.245302 9720 heartbeater.cc:347] Connected to a master server at 127.8.158.126:33913
I20260929 19:02:57.245427 9720 heartbeater.cc:464] Registering TS with master...
I20260929 19:02:57.245606 9720 heartbeater.cc:511] Master 127.8.158.126:33913 requested a full tablet report, sending...
I20260929 19:02:57.246055 9395 ts_manager.cc:194] Registered new tserver with Master: e0968d9fe5fe4bc7ac85411767291d78 (127.8.158.66:39215)
I20260929 19:02:57.246675 9395 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.66@KRBTEST.COM'} at 127.8.158.66:59911
WARNING: no policy specified for kudu/127.8.158.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab.
Entry for principal kudu/127.8.158.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab.
I20260929 19:02:57.269587 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.67:0
--local_ip_for_outbound_sockets=127.8.158.67
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:33913
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab
--principal=kudu/127.8.158.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:33785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:02:57.351164 9725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:57.351349 9725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:57.351372 9725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:57.352499 9725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:57.352540 9725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:57.352567 9725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.67
I20260929 19:02:57.354130 9725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:33785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab
--principal=kudu/127.8.158.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:33913
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.67
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:57.354364 9725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:57.354597 9725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:57.359584 9733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:57.359629 9730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:57.359612 9731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:57.359879 9725 server_base.cc:1034] running on GCE node
I20260929 19:02:57.360101 9725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:57.360283 9725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:57.361467 9725 hybrid_clock.cc:648] HybridClock initialized: now 1790708577361445 us; error 32 us; skew 500 ppm
Sep 29 19:02:57 dist-test-slave-nmsz krb5kdc[9350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708577, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:57.364091 9725 init.cc:377] Logged in from keytab as kudu/127.8.158.67@KRBTEST.COM (short username kudu)
I20260929 19:02:57.364569 9725 webserver.cc:460] Webserver started at http://127.8.158.67:46527/ using document root <none> and password file <none>
I20260929 19:02:57.364789 9725 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:57.364861 9725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:57.365007 9725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:57.366020 9725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-2/data/instance:
uuid: "b0634af099bc4226a125edfa75c2964c"
format_stamp: "Formatted at 2026-09-29 19:02:57 on dist-test-slave-nmsz"
server_key: "7729cc6c4827855f8ec359e389803697"
server_key_iv: "471feba2f8fbf2a08dc7843bd519f6ae"
server_key_version: "encryptionkey@0"
I20260929 19:02:57.366438 9725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance:
uuid: "b0634af099bc4226a125edfa75c2964c"
format_stamp: "Formatted at 2026-09-29 19:02:57 on dist-test-slave-nmsz"
server_key: "7729cc6c4827855f8ec359e389803697"
server_key_iv: "471feba2f8fbf2a08dc7843bd519f6ae"
server_key_version: "encryptionkey@0"
I20260929 19:02:57.367746 9725 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:02:57.368520 9740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:57.368716 9725 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:02:57.368822 9725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-2/wal
uuid: "b0634af099bc4226a125edfa75c2964c"
format_stamp: "Formatted at 2026-09-29 19:02:57 on dist-test-slave-nmsz"
server_key: "7729cc6c4827855f8ec359e389803697"
server_key_iv: "471feba2f8fbf2a08dc7843bd519f6ae"
server_key_version: "encryptionkey@0"
I20260929 19:02:57.368904 9725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:57.390686 9725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:57.392683 9725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:57.392838 9725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:57.393051 9725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:02:57.393358 9725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:02:57.393417 9725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:57.393467 9725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:02:57.393494 9725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:57.399345 9725 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.67:44325
I20260929 19:02:57.399381 9853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.67:44325 every 8 connection(s)
I20260929 19:02:57.399718 9725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
I20260929 19:02:57.404644 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 9725
I20260929 19:02:57.404742 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance
I20260929 19:02:57.404920 8825 external_mini_cluster.cc:1428] Setting key 5d03e646620daf75a4e973c9a3aa1cbd
Sep 29 19:02:57 dist-test-slave-nmsz krb5kdc[9350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708577, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.67@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:02:57.409341 9434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:57.400479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:59629 (local address 127.8.158.126:33913)
0929 19:02:57.400541 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:57.400544 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:57.400737 (+ 193us) server_negotiation.cc:407] Connection header received
0929 19:02:57.401658 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:57.401659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:57.401669 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:57.401724 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:57.402277 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:57.402831 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:57.403812 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:57.404007 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:57.404061 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:57.406674 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:57.406676 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:57.406676 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:57.406682 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:57.408085 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:57.408507 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:57.408508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:57.408508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:57.408558 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:57.408771 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:57.408772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:57.408772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:57.408876 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:57.408929 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:57.409187 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 19:02:57.409243 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 19:02:57.409499 9856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:57.400464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:33913 (local address 127.8.158.67:59629)
0929 19:02:57.400616 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:57.400650 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:57.401563 (+ 913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:57.401776 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:57.401779 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:57.402010 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:57.402194 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:57.402198 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:57.402913 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:57.402914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:57.403736 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:57.403738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:57.404083 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:57.404084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:57.404181 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:57.404825 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:02:57.404834 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:57.406583 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:57.408181 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:57.408186 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:57.408188 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:57.408427 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:57.408620 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:57.408620 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:57.408621 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:57.408716 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:57.408950 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:57.408955 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:57.409066 (+ 111us) client_negotiation.cc:769] Sending connection context
0929 19:02:57.409164 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 19:02:57.409273 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":92,"threads_started":1}
I20260929 19:02:57.410005 9854 heartbeater.cc:347] Connected to a master server at 127.8.158.126:33913
I20260929 19:02:57.410097 9854 heartbeater.cc:464] Registering TS with master...
I20260929 19:02:57.410223 9854 heartbeater.cc:511] Master 127.8.158.126:33913 requested a full tablet report, sending...
I20260929 19:02:57.410597 9395 ts_manager.cc:194] Registered new tserver with Master: b0634af099bc4226a125edfa75c2964c (127.8.158.67:44325)
I20260929 19:02:57.410925 9395 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.67@KRBTEST.COM'} at 127.8.158.67:59629
I20260929 19:02:57.416798 8825 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:02:57.424540 9434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:57.418177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60102 (local address 127.8.158.126:33913)
0929 19:02:57.418226 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:57.418228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:57.418432 (+ 204us) server_negotiation.cc:407] Connection header received
0929 19:02:57.418454 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:57.418455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:57.418462 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:57.418511 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:57.419021 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:57.419513 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:57.420287 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:57.420465 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:57.420521 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:57.421658 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:57.421659 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:57.421660 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:57.421665 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:57.423211 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:57.423646 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:57.423647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:57.423648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:57.423708 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:57.423926 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:57.423927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:57.423928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:57.424132 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:57.424248 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:57.424377 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 19:02:57.424460 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 19:02:57.426497 9395 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60102:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:02:57.427942 9395 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:02:57.434402 9865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:57.430162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.67:44325 (local address 127.0.0.1:46030)
0929 19:02:57.430658 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:57.430668 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:57.430733 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:57.430949 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:57.430950 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:57.430951 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:57.430991 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:57.430993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:57.432248 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:57.432249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:57.433409 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:57.433411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:57.434062 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:57.434063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:57.434142 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:57.434157 (+ 15us) client_negotiation.cc:769] Sending connection context
0929 19:02:57.434252 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 19:02:57.434306 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"mutex_wait_us":33,"spinlock_wait_cycles":768,"thread_start_us":74,"threads_started":1}
I20260929 19:02:57.434418 9868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:57.430292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46030 (local address 127.8.158.67:44325)
0929 19:02:57.430473 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:57.430478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:57.430721 (+ 243us) server_negotiation.cc:407] Connection header received
0929 19:02:57.430786 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:57.430788 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:57.430804 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:57.430871 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:57.431061 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:57.432142 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:57.433474 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:57.433962 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:57.434050 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:57.434106 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:57.434270 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 19:02:57.434352 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":81,"threads_started":1}
I20260929 19:02:57.434602 9870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:57.430488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53952 (local address 127.8.158.65:42511)
0929 19:02:57.430645 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:57.430649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:57.430861 (+ 212us) server_negotiation.cc:407] Connection header received
0929 19:02:57.430953 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:57.430955 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:57.430970 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:57.431045 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:57.431263 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:57.431919 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:57.433410 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:57.434086 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:57.434144 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:57.434194 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:57.434489 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 19:02:57.434526 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":92,"threads_started":1}
I20260929 19:02:57.434715 9867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:57.430253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57346 (local address 127.8.158.66:39215)
0929 19:02:57.430553 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:57.430559 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:57.430594 (+ 35us) server_negotiation.cc:407] Connection header received
0929 19:02:57.430695 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:57.430698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:57.430713 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:57.430784 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:57.431039 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:57.432154 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:57.433358 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:57.434071 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:57.434144 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:57.434194 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:57.434584 (+ 390us) server_negotiation.cc:299] Negotiation successful
0929 19:02:57.434632 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":111,"threads_started":1}
I20260929 19:02:57.434828 9869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:57.430387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:42511 (local address 127.0.0.1:53952)
0929 19:02:57.430795 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:57.430802 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:57.430861 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:57.431128 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:57.431129 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:57.431131 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:57.431184 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:57.431186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:57.432022 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:57.432024 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:57.433334 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:57.433337 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:57.434230 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:57.434231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:57.434340 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:57.434346 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 19:02:57.434398 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 19:02:57.434445 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":7,"thread_start_us":70,"threads_started":1}
I20260929 19:02:57.434522 9866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:57.430162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.66:39215 (local address 127.0.0.1:57346)
0929 19:02:57.430499 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:57.430539 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:57.430617 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:57.430875 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:57.430878 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:57.430881 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:57.430953 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:57.430954 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:57.432275 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:57.432275 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:57.433277 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:57.433279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:57.434232 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:57.434232 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:57.434339 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:57.434347 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 19:02:57.434407 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 19:02:57.434445 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":49,"thread_start_us":62,"threads_started":1}
I20260929 19:02:57.435523 9788 tablet_service.cc:1467] Processing CreateTablet for tablet 0563501bd7a0404fa3c28674c2f3c414 (DEFAULT_TABLE table=test-table [id=372d9f840d354120b00b4318eca856d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:02:57.435643 9520 tablet_service.cc:1467] Processing CreateTablet for tablet 0563501bd7a0404fa3c28674c2f3c414 (DEFAULT_TABLE table=test-table [id=372d9f840d354120b00b4318eca856d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:02:57.435643 9654 tablet_service.cc:1467] Processing CreateTablet for tablet 0563501bd7a0404fa3c28674c2f3c414 (DEFAULT_TABLE table=test-table [id=372d9f840d354120b00b4318eca856d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:02:57.435877 9788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0563501bd7a0404fa3c28674c2f3c414. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:57.435966 9520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0563501bd7a0404fa3c28674c2f3c414. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:57.435966 9654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0563501bd7a0404fa3c28674c2f3c414. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:57.438462 9871 tablet_bootstrap.cc:492] T 0563501bd7a0404fa3c28674c2f3c414 P b0634af099bc4226a125edfa75c2964c: Bootstrap starting.
I20260929 19:02:57.439090 9871 tablet_bootstrap.cc:654] T 0563501bd7a0404fa3c28674c2f3c414 P b0634af099bc4226a125edfa75c2964c: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:57.439153 9873 tablet_bootstrap.cc:492] T 0563501bd7a0404fa3c28674c2f3c414 P ba4f93b763594b82b9884e440a60aad0: Bootstrap starting.
I20260929 19:02:57.439746 9871 log.cc:824] T 0563501bd7a0404fa3c28674c2f3c414 P b0634af099bc4226a125edfa75c2964c: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:57.439798 9873 tablet_bootstrap.cc:654] T 0563501bd7a0404fa3c28674c2f3c414 P ba4f93b763594b82b9884e440a60aad0: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:57.440351 9873 log.cc:824] T 0563501bd7a0404fa3c28674c2f3c414 P ba4f93b763594b82b9884e440a60aad0: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:57.440503 9871 tablet_bootstrap.cc:492] T 0563501bd7a0404fa3c28674c2f3c414 P b0634af099bc4226a125edfa75c2964c: No bootstrap required, opened a new log
I20260929 19:02:57.440577 9871 ts_tablet_manager.cc:1397] T 0563501bd7a0404fa3c28674c2f3c414 P b0634af099bc4226a125edfa75c2964c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:02:57.440675 9872 tablet_bootstrap.cc:492] T 0563501bd7a0404fa3c28674c2f3c414 P e0968d9fe5fe4bc7ac85411767291d78: Bootstrap starting.
I20260929 19:02:57.441064 9873 tablet_bootstrap.cc:492] T 0563501bd7a0404fa3c28674c2f3c414 P ba4f93b763594b82b9884e440a60aad0: No bootstrap required, opened a new log
I20260929 19:02:57.441146 9873 ts_tablet_manager.cc:1397] T 0563501bd7a0404fa3c28674c2f3c414 P ba4f93b763594b82b9884e440a60aad0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:02:57.441275 9872 tablet_bootstrap.cc:654] T 0563501bd7a0404fa3c28674c2f3c414 P e0968d9fe5fe4bc7ac85411767291d78: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:57.441819 9872 log.cc:824] T 0563501bd7a0404fa3c28674c2f3c414 P e0968d9fe5fe4bc7ac85411767291d78: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:57.442628 9872 tablet_bootstrap.cc:492] T 0563501bd7a0404fa3c28674c2f3c414 P e0968d9fe5fe4bc7ac85411767291d78: No bootstrap required, opened a new log
I20260929 19:02:57.442701 9872 ts_tablet_manager.cc:1397] T 0563501bd7a0404fa3c28674c2f3c414 P e0968d9fe5fe4bc7ac85411767291d78: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:02:57.442683 9871 raft_consensus.cc:348] T 0563501bd7a0404fa3c28674c2f3c414 P b0634af099bc4226a125edfa75c2964c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0968d9fe5fe4bc7ac85411767291d78" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 39215 } } peers { permanent_uuid: "ba4f93b763594b82b9884e440a60aad0" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 42511 } } peers { permanent_uuid: "b0634af099bc4226a125edfa75c2964c" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 44325 } }
I20260929 19:02:57.442854 9871 raft_consensus.cc:374] T 0563501bd7a0404fa3c28674c2f3c414 P b0634af099bc4226a125edfa75c2964c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:57.442898 9871 raft_consensus.cc:729] T 0563501bd7a0404fa3c28674c2f3c414 P b0634af099bc4226a125edfa75c2964c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0634af099bc4226a125edfa75c2964c, State: Initialized, Role: FOLLOWER
I20260929 19:02:57.443017 9871 consensus_queue.cc:260] T 0563501bd7a0404fa3c28674c2f3c414 P b0634af099bc4226a125edfa75c2964c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0968d9fe5fe4bc7ac85411767291d78" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 39215 } } peers { permanent_uuid: "ba4f93b763594b82b9884e440a60aad0" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 42511 } } peers { permanent_uuid: "b0634af099bc4226a125edfa75c2964c" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 44325 } }
I20260929 19:02:57.443266 9871 ts_tablet_manager.cc:1428] T 0563501bd7a0404fa3c28674c2f3c414 P b0634af099bc4226a125edfa75c2964c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 19:02:57.443207 9873 raft_consensus.cc:348] T 0563501bd7a0404fa3c28674c2f3c414 P ba4f93b763594b82b9884e440a60aad0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0968d9fe5fe4bc7ac85411767291d78" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 39215 } } peers { permanent_uuid: "ba4f93b763594b82b9884e440a60aad0" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 42511 } } peers { permanent_uuid: "b0634af099bc4226a125edfa75c2964c" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 44325 } }
I20260929 19:02:57.443378 9873 raft_consensus.cc:374] T 0563501bd7a0404fa3c28674c2f3c414 P ba4f93b763594b82b9884e440a60aad0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:57.443379 9854 heartbeater.cc:503] Master 127.8.158.126:33913 was elected leader, sending a full tablet report...
I20260929 19:02:57.443424 9873 raft_consensus.cc:729] T 0563501bd7a0404fa3c28674c2f3c414 P ba4f93b763594b82b9884e440a60aad0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba4f93b763594b82b9884e440a60aad0, State: Initialized, Role: FOLLOWER
I20260929 19:02:57.443540 9873 consensus_queue.cc:260] T 0563501bd7a0404fa3c28674c2f3c414 P ba4f93b763594b82b9884e440a60aad0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0968d9fe5fe4bc7ac85411767291d78" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 39215 } } peers { permanent_uuid: "ba4f93b763594b82b9884e440a60aad0" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 42511 } } peers { permanent_uuid: "b0634af099bc4226a125edfa75c2964c" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 44325 } }
I20260929 19:02:57.443791 9873 ts_tablet_manager.cc:1428] T 0563501bd7a0404fa3c28674c2f3c414 P ba4f93b763594b82b9884e440a60aad0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 19:02:57.443920 9586 heartbeater.cc:503] Master 127.8.158.126:33913 was elected leader, sending a full tablet report...
I20260929 19:02:57.444530 9872 raft_consensus.cc:348] T 0563501bd7a0404fa3c28674c2f3c414 P e0968d9fe5fe4bc7ac85411767291d78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0968d9fe5fe4bc7ac85411767291d78" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 39215 } } peers { permanent_uuid: "ba4f93b763594b82b9884e440a60aad0" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 42511 } } peers { permanent_uuid: "b0634af099bc4226a125edfa75c2964c" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 44325 } }
I20260929 19:02:57.444657 9872 raft_consensus.cc:374] T 0563501bd7a0404fa3c28674c2f3c414 P e0968d9fe5fe4bc7ac85411767291d78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:57.444679 9872 raft_consensus.cc:729] T 0563501bd7a0404fa3c28674c2f3c414 P e0968d9fe5fe4bc7ac85411767291d78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0968d9fe5fe4bc7ac85411767291d78, State: Initialized, Role: FOLLOWER
I20260929 19:02:57.444772 9872 consensus_queue.cc:260] T 0563501bd7a0404fa3c28674c2f3c414 P e0968d9fe5fe4bc7ac85411767291d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0968d9fe5fe4bc7ac85411767291d78" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 39215 } } peers { permanent_uuid: "ba4f93b763594b82b9884e440a60aad0" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 42511 } } peers { permanent_uuid: "b0634af099bc4226a125edfa75c2964c" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 44325 } }
I20260929 19:02:57.445021 9872 ts_tablet_manager.cc:1428] T 0563501bd7a0404fa3c28674c2f3c414 P e0968d9fe5fe4bc7ac85411767291d78: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260929 19:02:57.445034 9720 heartbeater.cc:503] Master 127.8.158.126:33913 was elected leader, sending a full tablet report...
I20260929 19:02:57.458302 9877 raft_consensus.cc:482] T 0563501bd7a0404fa3c28674c2f3c414 P b0634af099bc4226a125edfa75c2964c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:02:57.458489 9877 raft_consensus.cc:504] T 0563501bd7a0404fa3c28674c2f3c414 P b0634af099bc4226a125edfa75c2964c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0968d9fe5fe4bc7ac85411767291d78" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 39215 } } peers { permanent_uuid: "ba4f93b763594b82b9884e440a60aad0" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 42511 } } peers { permanent_uuid: "b0634af099bc4226a125edfa75c2964c" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 44325 } }
I20260929 19:02:57.458747 9877 leader_election.cc:288] T 0563501bd7a0404fa3c28674c2f3c414 P b0634af099bc4226a125edfa75c2964c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0968d9fe5fe4bc7ac85411767291d78 (127.8.158.66:39215), ba4f93b763594b82b9884e440a60aad0 (127.8.158.65:42511)
I20260929 19:02:57.462867 9870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:57.459221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:39217 (local address 127.8.158.65:42511)
0929 19:02:57.459261 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:57.459263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:57.459311 (+ 48us) server_negotiation.cc:407] Connection header received
0929 19:02:57.459372 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:57.459373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:57.459380 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:57.459420 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:57.459565 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:57.460332 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:57.461831 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:57.462514 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:57.462583 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:57.462592 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:57.462757 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 19:02:57.462796 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 19:02:57.462870 9867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:57.459101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:36829 (local address 127.8.158.66:39215)
0929 19:02:57.459156 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:57.459159 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:57.459171 (+ 12us) server_negotiation.cc:407] Connection header received
0929 19:02:57.459197 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:57.459198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:57.459208 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:57.459278 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:57.459469 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:57.460314 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:57.461844 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:57.462487 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:57.462576 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:57.462585 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:57.462760 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 19:02:57.462797 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 19:02:57.462900 9856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:57.459024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.66:39215 (local address 127.8.158.67:36829)
0929 19:02:57.459079 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:57.459091 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:57.459132 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:57.459334 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:57.459335 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:57.459336 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:57.459398 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:57.459399 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:57.460430 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:57.460431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:57.461724 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:57.461727 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:57.462588 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:57.462589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:57.462670 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:57.462674 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:02:57.462736 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 19:02:57.462783 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":70}
I20260929 19:02:57.462908 9880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:57.459135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:42511 (local address 127.8.158.67:39217)
0929 19:02:57.459261 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:57.459270 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:57.459318 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:57.459432 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:57.459433 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:57.459435 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:57.459499 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:57.459500 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:57.460430 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:57.460431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:57.461724 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:57.461727 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:57.462588 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:57.462589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:57.462670 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:57.462674 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:02:57.462736 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 19:02:57.462783 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":37,"thread_start_us":89,"threads_started":1}
I20260929 19:02:57.463246 9674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0563501bd7a0404fa3c28674c2f3c414" candidate_uuid: "b0634af099bc4226a125edfa75c2964c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0968d9fe5fe4bc7ac85411767291d78" is_pre_election: true
I20260929 19:02:57.463426 9674 raft_consensus.cc:2454] T 0563501bd7a0404fa3c28674c2f3c414 P e0968d9fe5fe4bc7ac85411767291d78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0634af099bc4226a125edfa75c2964c in term 0.
I20260929 19:02:57.463419 9540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0563501bd7a0404fa3c28674c2f3c414" candidate_uuid: "b0634af099bc4226a125edfa75c2964c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba4f93b763594b82b9884e440a60aad0" is_pre_election: true
I20260929 19:02:57.463560 9540 raft_consensus.cc:2454] T 0563501bd7a0404fa3c28674c2f3c414 P ba4f93b763594b82b9884e440a60aad0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0634af099bc4226a125edfa75c2964c in term 0.
I20260929 19:02:57.463644 9741 leader_election.cc:302] T 0563501bd7a0404fa3c28674c2f3c414 P b0634af099bc4226a125edfa75c2964c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0634af099bc4226a125edfa75c2964c, e0968d9fe5fe4bc7ac85411767291d78; no voters:
I20260929 19:02:57.463775 9877 raft_consensus.cc:2781] T 0563501bd7a0404fa3c28674c2f3c414 P b0634af099bc4226a125edfa75c2964c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:02:57.463855 9877 raft_consensus.cc:482] T 0563501bd7a0404fa3c28674c2f3c414 P b0634af099bc4226a125edfa75c2964c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:02:57.463892 9877 raft_consensus.cc:3037] T 0563501bd7a0404fa3c28674c2f3c414 P b0634af099bc4226a125edfa75c2964c [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:57.464587 9877 raft_consensus.cc:504] T 0563501bd7a0404fa3c28674c2f3c414 P b0634af099bc4226a125edfa75c2964c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0968d9fe5fe4bc7ac85411767291d78" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 39215 } } peers { permanent_uuid: "ba4f93b763594b82b9884e440a60aad0" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 42511 } } peers { permanent_uuid: "b0634af099bc4226a125edfa75c2964c" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 44325 } }
I20260929 19:02:57.464720 9877 leader_election.cc:288] T 0563501bd7a0404fa3c28674c2f3c414 P b0634af099bc4226a125edfa75c2964c [CANDIDATE]: Term 1 election: Requested vote from peers e0968d9fe5fe4bc7ac85411767291d78 (127.8.158.66:39215), ba4f93b763594b82b9884e440a60aad0 (127.8.158.65:42511)
I20260929 19:02:57.464885 9674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0563501bd7a0404fa3c28674c2f3c414" candidate_uuid: "b0634af099bc4226a125edfa75c2964c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0968d9fe5fe4bc7ac85411767291d78"
I20260929 19:02:57.464928 9540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0563501bd7a0404fa3c28674c2f3c414" candidate_uuid: "b0634af099bc4226a125edfa75c2964c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba4f93b763594b82b9884e440a60aad0"
I20260929 19:02:57.464982 9540 raft_consensus.cc:3037] T 0563501bd7a0404fa3c28674c2f3c414 P ba4f93b763594b82b9884e440a60aad0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:57.465000 9674 raft_consensus.cc:3037] T 0563501bd7a0404fa3c28674c2f3c414 P e0968d9fe5fe4bc7ac85411767291d78 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:57.465859 9540 raft_consensus.cc:2454] T 0563501bd7a0404fa3c28674c2f3c414 P ba4f93b763594b82b9884e440a60aad0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0634af099bc4226a125edfa75c2964c in term 1.
I20260929 19:02:57.465879 9674 raft_consensus.cc:2454] T 0563501bd7a0404fa3c28674c2f3c414 P e0968d9fe5fe4bc7ac85411767291d78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0634af099bc4226a125edfa75c2964c in term 1.
I20260929 19:02:57.466023 9741 leader_election.cc:302] T 0563501bd7a0404fa3c28674c2f3c414 P b0634af099bc4226a125edfa75c2964c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0634af099bc4226a125edfa75c2964c, ba4f93b763594b82b9884e440a60aad0; no voters:
I20260929 19:02:57.466125 9877 raft_consensus.cc:2781] T 0563501bd7a0404fa3c28674c2f3c414 P b0634af099bc4226a125edfa75c2964c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:02:57.466249 9877 raft_consensus.cc:686] T 0563501bd7a0404fa3c28674c2f3c414 P b0634af099bc4226a125edfa75c2964c [term 1 LEADER]: Becoming Leader. State: Replica: b0634af099bc4226a125edfa75c2964c, State: Running, Role: LEADER
I20260929 19:02:57.466373 9877 consensus_queue.cc:237] T 0563501bd7a0404fa3c28674c2f3c414 P b0634af099bc4226a125edfa75c2964c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0968d9fe5fe4bc7ac85411767291d78" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 39215 } } peers { permanent_uuid: "ba4f93b763594b82b9884e440a60aad0" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 42511 } } peers { permanent_uuid: "b0634af099bc4226a125edfa75c2964c" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 44325 } }
I20260929 19:02:57.467128 9393 catalog_manager.cc:5495] T 0563501bd7a0404fa3c28674c2f3c414 P b0634af099bc4226a125edfa75c2964c reported cstate change: term changed from 0 to 1, leader changed from <none> to b0634af099bc4226a125edfa75c2964c (127.8.158.67). New cstate: current_term: 1 leader_uuid: "b0634af099bc4226a125edfa75c2964c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0968d9fe5fe4bc7ac85411767291d78" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 39215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba4f93b763594b82b9884e440a60aad0" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 42511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0634af099bc4226a125edfa75c2964c" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 44325 } health_report { overall_health: HEALTHY } } }
Sep 29 19:02:57 dist-test-slave-nmsz krb5kdc[9350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.158.65@KRBTEST.COM
I20260929 19:02:57.483518 9870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:57.476079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53966 (local address 127.8.158.65:42511)
0929 19:02:57.476136 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:57.476138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:57.476288 (+ 150us) server_negotiation.cc:407] Connection header received
0929 19:02:57.476332 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:57.476333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:57.476342 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:57.476393 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:57.476921 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:57.477385 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:57.478055 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:57.478224 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:57.478275 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:57.480650 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:57.480657 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:57.480658 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:57.480664 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:57.482295 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:57.482720 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:57.482723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:57.482723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:57.482779 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:57.482999 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:57.483000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:57.483000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:57.483132 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:57.483289 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:57.483329 (+ 40us) server_negotiation.cc:299] Negotiation successful
0929 19:02:57.483405 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260929 19:02:57.484974 9721 tablet.cc:2369] T 0563501bd7a0404fa3c28674c2f3c414 P e0968d9fe5fe4bc7ac85411767291d78: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 19:02:57 dist-test-slave-nmsz krb5kdc[9350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708577, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 19:02:57 dist-test-slave-nmsz krb5kdc[9350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708577, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.158.65@KRBTEST.COM
I20260929 19:02:57.504899 9870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:57.498178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53972 (local address 127.8.158.65:42511)
0929 19:02:57.498223 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:57.498225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:57.498288 (+ 63us) server_negotiation.cc:407] Connection header received
0929 19:02:57.498387 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:57.498388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:57.498396 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:57.498457 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:57.498977 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:57.499419 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:57.500106 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:57.500277 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:57.500325 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:57.502548 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:57.502550 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:57.502550 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:57.502556 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:57.503907 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:57.504232 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:57.504233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:57.504233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:57.504279 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:57.504473 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:57.504474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:57.504474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:57.504555 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:57.504624 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:57.504785 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 19:02:57.504838 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Sep 29 19:02:57 dist-test-slave-nmsz krb5kdc[9350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 19:02:57 dist-test-slave-nmsz krb5kdc[9350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.158.65@KRBTEST.COM
I20260929 19:02:57.525995 9870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:57.519117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53974 (local address 127.8.158.65:42511)
0929 19:02:57.519190 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:57.519192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:57.519206 (+ 14us) server_negotiation.cc:407] Connection header received
0929 19:02:57.519265 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:57.519266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:57.519275 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:57.519334 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:57.519870 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:57.520394 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:57.521090 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:57.521265 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:57.521325 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:57.523801 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:57.523803 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:57.523803 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:57.523809 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:57.525020 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:57.525336 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:57.525337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:57.525337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:57.525384 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:57.525556 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:57.525557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:57.525557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:57.525637 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:57.525706 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:57.525869 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 19:02:57.525931 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 19:02:57.526480 9500 tablet_service.cc:2908] Rejecting scan request for tablet 0563501bd7a0404fa3c28674c2f3c414: Uninitialized: clean time has not yet been initialized
I20260929 19:02:57.527571 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 9457
I20260929 19:02:57.532116 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 9591
I20260929 19:02:57.536966 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 9725
I20260929 19:02:57.541297 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 9363
2026-09-29T19:02:57Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1943 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:02:57 dist-test-slave-nmsz krb5kdc[9895](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:02:57 dist-test-slave-nmsz krb5kdc[9895](info): set up 2 sockets
Sep 29 19:02:57 dist-test-slave-nmsz krb5kdc[9895](info): commencing operation
krb5kdc: starting...
W20260929 19:02:58.614044 8825 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Sep 29 19:02:58 dist-test-slave-nmsz krb5kdc[9895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:02:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:02:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.158.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
I20260929 19:02:58.725961 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:45077
--webserver_interface=127.8.158.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:42801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:45077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:02:58.803696 9908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:58.803929 9908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:58.803962 9908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:58.805088 9908 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:02:58.805161 9908 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 19:02:58.805183 9908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:58.805205 9908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:58.805234 9908 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:02:58.805265 9908 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:02:58.806725 9908 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:42801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:45077
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:45077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.158.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:58.806984 9908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:58.807199 9908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:58.812090 9916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:58.812100 9914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:58.812065 9913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:58.812336 9908 server_base.cc:1034] running on GCE node
I20260929 19:02:58.812548 9908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:58.812826 9908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:58.813995 9908 hybrid_clock.cc:648] HybridClock initialized: now 1790708578813997 us; error 46 us; skew 500 ppm
Sep 29 19:02:58 dist-test-slave-nmsz krb5kdc[9895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708578, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:58.828174 9908 init.cc:377] Logged in from keytab as kudu/127.8.158.126@KRBTEST.COM (short username kudu)
I20260929 19:02:58.828624 9908 webserver.cc:460] Webserver started at http://127.8.158.126:44991/ using document root <none> and password file <none>
I20260929 19:02:58.828850 9908 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:58.828918 9908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:58.829078 9908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:58.830154 9908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "869266291b584c3583320a409b4e2c9c"
format_stamp: "Formatted at 2026-09-29 19:02:58 on dist-test-slave-nmsz"
server_key: "0a4669b2672b4379aba8bbc5df3895e7"
server_key_iv: "63e5013df86cbbcda6f0c067f0592f9f"
server_key_version: "encryptionkey@0"
I20260929 19:02:58.830561 9908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "869266291b584c3583320a409b4e2c9c"
format_stamp: "Formatted at 2026-09-29 19:02:58 on dist-test-slave-nmsz"
server_key: "0a4669b2672b4379aba8bbc5df3895e7"
server_key_iv: "63e5013df86cbbcda6f0c067f0592f9f"
server_key_version: "encryptionkey@0"
I20260929 19:02:58.831897 9908 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:02:58.832640 9923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:58.832821 9908 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:02:58.832921 9908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "869266291b584c3583320a409b4e2c9c"
format_stamp: "Formatted at 2026-09-29 19:02:58 on dist-test-slave-nmsz"
server_key: "0a4669b2672b4379aba8bbc5df3895e7"
server_key_iv: "63e5013df86cbbcda6f0c067f0592f9f"
server_key_version: "encryptionkey@0"
I20260929 19:02:58.833011 9908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:58.851061 9908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:58.853396 9908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:58.853564 9908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:58.858454 9908 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.126:45077
I20260929 19:02:58.858539 9985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.126:45077 every 8 connection(s)
I20260929 19:02:58.858855 9908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:02:58.859606 9986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:58.860608 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 9908
I20260929 19:02:58.860714 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:02:58.860888 8825 external_mini_cluster.cc:1428] Setting key 206c43984d016953818291eff512bfcd
I20260929 19:02:58.862396 9986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 869266291b584c3583320a409b4e2c9c: Bootstrap starting.
I20260929 19:02:58.862991 9986 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 869266291b584c3583320a409b4e2c9c: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:58.863327 9986 log.cc:824] T 00000000000000000000000000000000 P 869266291b584c3583320a409b4e2c9c: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:58.864058 9986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 869266291b584c3583320a409b4e2c9c: No bootstrap required, opened a new log
I20260929 19:02:58.866297 9986 raft_consensus.cc:348] T 00000000000000000000000000000000 P 869266291b584c3583320a409b4e2c9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869266291b584c3583320a409b4e2c9c" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 45077 } }
I20260929 19:02:58.866513 9986 raft_consensus.cc:374] T 00000000000000000000000000000000 P 869266291b584c3583320a409b4e2c9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:58.866557 9986 raft_consensus.cc:729] T 00000000000000000000000000000000 P 869266291b584c3583320a409b4e2c9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869266291b584c3583320a409b4e2c9c, State: Initialized, Role: FOLLOWER
I20260929 19:02:58.866683 9986 consensus_queue.cc:260] T 00000000000000000000000000000000 P 869266291b584c3583320a409b4e2c9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869266291b584c3583320a409b4e2c9c" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 45077 } }
I20260929 19:02:58.866768 9986 raft_consensus.cc:388] T 00000000000000000000000000000000 P 869266291b584c3583320a409b4e2c9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:02:58.866801 9986 raft_consensus.cc:482] T 00000000000000000000000000000000 P 869266291b584c3583320a409b4e2c9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:02:58.866843 9986 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 869266291b584c3583320a409b4e2c9c [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:58.867550 9986 raft_consensus.cc:504] T 00000000000000000000000000000000 P 869266291b584c3583320a409b4e2c9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869266291b584c3583320a409b4e2c9c" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 45077 } }
I20260929 19:02:58.867684 9986 leader_election.cc:302] T 00000000000000000000000000000000 P 869266291b584c3583320a409b4e2c9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 869266291b584c3583320a409b4e2c9c; no voters:
I20260929 19:02:58.867836 9986 leader_election.cc:288] T 00000000000000000000000000000000 P 869266291b584c3583320a409b4e2c9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:02:58.867857 9991 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 869266291b584c3583320a409b4e2c9c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:02:58.867959 9991 raft_consensus.cc:686] T 00000000000000000000000000000000 P 869266291b584c3583320a409b4e2c9c [term 1 LEADER]: Becoming Leader. State: Replica: 869266291b584c3583320a409b4e2c9c, State: Running, Role: LEADER
I20260929 19:02:58.868068 9991 consensus_queue.cc:237] T 00000000000000000000000000000000 P 869266291b584c3583320a409b4e2c9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869266291b584c3583320a409b4e2c9c" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 45077 } }
I20260929 19:02:58.868098 9986 sys_catalog.cc:564] T 00000000000000000000000000000000 P 869266291b584c3583320a409b4e2c9c [sys.catalog]: configured and running, proceeding with master startup.
Sep 29 19:02:58 dist-test-slave-nmsz krb5kdc[9895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:02:58.868402 9992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 869266291b584c3583320a409b4e2c9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "869266291b584c3583320a409b4e2c9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869266291b584c3583320a409b4e2c9c" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 45077 } } }
I20260929 19:02:58.868491 9992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 869266291b584c3583320a409b4e2c9c [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:58.868640 9993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 869266291b584c3583320a409b4e2c9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 869266291b584c3583320a409b4e2c9c. Latest consensus state: current_term: 1 leader_uuid: "869266291b584c3583320a409b4e2c9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869266291b584c3583320a409b4e2c9c" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 45077 } } }
I20260929 19:02:58.868710 9993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 869266291b584c3583320a409b4e2c9c [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:58.869053 9995 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:02:58.869874 9995 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:02:58.871833 9995 catalog_manager.cc:1347] Generated new cluster ID: deb1bd98da1f4efa8144cd4d1886aa89
I20260929 19:02:58.871889 9995 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:02:58.873138 9989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:58.861434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59450 (local address 127.8.158.126:45077)
0929 19:02:58.861620 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:58.861629 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:58.861660 (+ 31us) server_negotiation.cc:407] Connection header received
0929 19:02:58.862154 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:58.862162 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:58.862219 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:58.862388 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:58.863083 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:58.864433 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:58.865210 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:58.865814 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:58.865938 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:58.869024 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:58.869028 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:58.869028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:58.869040 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:58.871199 (+ 2159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:58.871682 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:58.871684 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:58.871685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:58.871796 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:58.872005 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:58.872006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:58.872006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:58.872289 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:58.872424 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:58.872810 (+ 386us) server_negotiation.cc:299] Negotiation successful
0929 19:02:58.872931 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":83,"threads_started":1}
I20260929 19:02:58.878223 9995 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:02:58.878937 9995 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:02:58.894958 9995 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 869266291b584c3583320a409b4e2c9c: Generated new TSK 0
I20260929 19:02:58.895138 9995 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.158.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab.
Entry for principal kudu/127.8.158.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab.
I20260929 19:02:58.928634 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.65:0
--local_ip_for_outbound_sockets=127.8.158.65
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:45077
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab
--principal=kudu/127.8.158.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:42801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:02:59.013329 10012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:59.013506 10012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:59.013530 10012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:59.014752 10012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:59.014791 10012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:59.014813 10012 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 19:02:59.014832 10012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.65
I20260929 19:02:59.016217 10012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:42801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab
--principal=kudu/127.8.158.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.158.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.158.126:45077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.65
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:59.016453 10012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:59.016671 10012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:59.021508 10020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:59.021497 10017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:59.021484 10018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:59.021739 10012 server_base.cc:1034] running on GCE node
I20260929 19:02:59.021929 10012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:59.022105 10012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:59.023291 10012 hybrid_clock.cc:648] HybridClock initialized: now 1790708579023279 us; error 26 us; skew 500 ppm
Sep 29 19:02:59 dist-test-slave-nmsz krb5kdc[9895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708579, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:59.025799 10012 init.cc:377] Logged in from keytab as kudu/127.8.158.65@KRBTEST.COM (short username kudu)
I20260929 19:02:59.026206 10012 webserver.cc:460] Webserver started at http://127.8.158.65:42177/ using document root <none> and password file <none>
I20260929 19:02:59.026508 10012 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:59.026556 10012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:59.026719 10012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:59.027688 10012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-0/data/instance:
uuid: "bfeeef88c11247529376decbf76640f4"
format_stamp: "Formatted at 2026-09-29 19:02:59 on dist-test-slave-nmsz"
server_key: "4119ba1fad9b742a1430612abf76e40b"
server_key_iv: "fac5e1ba951c185b913d5cce406666ff"
server_key_version: "encryptionkey@0"
I20260929 19:02:59.028026 10012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance:
uuid: "bfeeef88c11247529376decbf76640f4"
format_stamp: "Formatted at 2026-09-29 19:02:59 on dist-test-slave-nmsz"
server_key: "4119ba1fad9b742a1430612abf76e40b"
server_key_iv: "fac5e1ba951c185b913d5cce406666ff"
server_key_version: "encryptionkey@0"
I20260929 19:02:59.029382 10012 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:02:59.030148 10027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:59.030403 10012 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:02:59.030473 10012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-0/wal
uuid: "bfeeef88c11247529376decbf76640f4"
format_stamp: "Formatted at 2026-09-29 19:02:59 on dist-test-slave-nmsz"
server_key: "4119ba1fad9b742a1430612abf76e40b"
server_key_iv: "fac5e1ba951c185b913d5cce406666ff"
server_key_version: "encryptionkey@0"
I20260929 19:02:59.030560 10012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:59.053233 10012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:59.055177 10012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:59.055363 10012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:59.055842 10012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:02:59.055877 10012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:59.055899 10012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:02:59.055950 10012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 19:02:59 dist-test-slave-nmsz krb5kdc[9895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708579, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:02:59.062913 10012 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.65:45603
I20260929 19:02:59.063004 10142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.65:45603 every 8 connection(s)
I20260929 19:02:59.063301 10012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
I20260929 19:02:59.065651 9989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:59.056387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:58961 (local address 127.8.158.126:45077)
0929 19:02:59.056444 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:59.056446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:59.056645 (+ 199us) server_negotiation.cc:407] Connection header received
0929 19:02:59.057197 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:59.057199 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:59.057213 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:59.057285 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:59.058184 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.059056 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.060178 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.060537 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.060618 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:59.063003 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:59.063005 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:59.063005 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:59.063012 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:59.064580 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:59.064924 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:59.064925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:59.064925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:59.064973 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:59.065171 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:59.065172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:59.065172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:59.065262 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:59.065310 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:59.065528 (+ 218us) server_negotiation.cc:299] Negotiation successful
0929 19:02:59.065587 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 19:02:59.065811 10036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:59.056366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:45077 (local address 127.8.158.65:58961)
0929 19:02:59.056526 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:59.056561 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:59.057047 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:59.057366 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:59.057371 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:59.057771 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:59.058068 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:59.058075 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.059181 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.059182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:59.060083 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:59.060087 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.060638 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.060639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:59.060790 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:59.061327 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:02:59.061340 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:59.062887 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:59.064648 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:59.064654 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:59.064655 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:59.064855 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:59.065030 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:59.065031 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:59.065031 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:59.065122 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:59.065316 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:59.065320 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:59.065419 (+ 99us) client_negotiation.cc:769] Sending connection context
0929 19:02:59.065516 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 19:02:59.065624 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":80,"threads_started":1}
I20260929 19:02:59.066249 10143 heartbeater.cc:347] Connected to a master server at 127.8.158.126:45077
I20260929 19:02:59.066345 10143 heartbeater.cc:464] Registering TS with master...
I20260929 19:02:59.066516 10143 heartbeater.cc:511] Master 127.8.158.126:45077 requested a full tablet report, sending...
I20260929 19:02:59.067020 9940 ts_manager.cc:194] Registered new tserver with Master: bfeeef88c11247529376decbf76640f4 (127.8.158.65:45603)
I20260929 19:02:59.067680 9940 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.65@KRBTEST.COM'} at 127.8.158.65:58961
I20260929 19:02:59.072942 9989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:59.067284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:54477 (local address 127.8.158.126:45077)
0929 19:02:59.067883 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:59.067885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:59.067894 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:02:59.067937 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:59.067938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:59.067947 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:59.067995 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:59.068388 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.068835 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.069612 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.069785 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.069844 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:59.070671 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:59.070672 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:59.070672 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:59.070677 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:59.072032 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:59.072388 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:59.072389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:59.072389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:59.072435 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:59.072612 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:59.072612 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:59.072612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:59.072692 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:59.072734 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:59.072842 (+ 108us) server_negotiation.cc:299] Negotiation successful
0929 19:02:59.072890 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583}
I20260929 19:02:59.073043 10149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:59.067226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:45077 (local address 127.8.158.65:54477)
0929 19:02:59.067332 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:59.067342 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:59.067391 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:59.068051 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:59.068052 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:59.068263 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:59.068313 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:59.068315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.068993 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.068994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:59.069537 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:59.069539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.069854 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.069855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:59.069910 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:59.070300 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:02:59.070305 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:59.070601 (+ 296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:59.072116 (+ 1515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:59.072117 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:59.072117 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:59.072323 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:59.072519 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:59.072520 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:59.072520 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:59.072569 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:59.072744 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:59.072744 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:59.072778 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 19:02:59.072829 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 19:02:59.072948 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":61,"threads_started":1}
I20260929 19:02:59.073415 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 10012
I20260929 19:02:59.073515 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance
I20260929 19:02:59.073688 8825 external_mini_cluster.cc:1428] Setting key 6b33903587b15e003e1a4b00955cce21
WARNING: no policy specified for kudu/127.8.158.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab.
Entry for principal kudu/127.8.158.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab.
I20260929 19:02:59.096434 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.66:0
--local_ip_for_outbound_sockets=127.8.158.66
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:45077
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab
--principal=kudu/127.8.158.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:42801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:02:59.173769 10152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:59.173955 10152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:59.173977 10152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:59.175115 10152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:59.175156 10152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:59.175176 10152 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 19:02:59.175220 10152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.66
I20260929 19:02:59.176600 10152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:42801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab
--principal=kudu/127.8.158.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.158.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.158.126:45077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.66
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:59.176824 10152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:59.177042 10152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:59.181808 10160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:59.181840 10157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:59.181815 10158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:59.181977 10152 server_base.cc:1034] running on GCE node
I20260929 19:02:59.182264 10152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:59.182510 10152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:59.183647 10152 hybrid_clock.cc:648] HybridClock initialized: now 1790708579183636 us; error 32 us; skew 500 ppm
Sep 29 19:02:59 dist-test-slave-nmsz krb5kdc[9895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708579, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:59.186008 10152 init.cc:377] Logged in from keytab as kudu/127.8.158.66@KRBTEST.COM (short username kudu)
I20260929 19:02:59.186502 10152 webserver.cc:460] Webserver started at http://127.8.158.66:36447/ using document root <none> and password file <none>
I20260929 19:02:59.186712 10152 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:59.186755 10152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:59.186896 10152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:59.187867 10152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-1/data/instance:
uuid: "40549454a29b43d1a62d6d9fbdc5a7cf"
format_stamp: "Formatted at 2026-09-29 19:02:59 on dist-test-slave-nmsz"
server_key: "7817a30a8f50c8c24007e54c07dbd6e5"
server_key_iv: "dd0483fa836555ad30e0170f6b16bb2d"
server_key_version: "encryptionkey@0"
I20260929 19:02:59.188207 10152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance:
uuid: "40549454a29b43d1a62d6d9fbdc5a7cf"
format_stamp: "Formatted at 2026-09-29 19:02:59 on dist-test-slave-nmsz"
server_key: "7817a30a8f50c8c24007e54c07dbd6e5"
server_key_iv: "dd0483fa836555ad30e0170f6b16bb2d"
server_key_version: "encryptionkey@0"
I20260929 19:02:59.189486 10152 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:02:59.190258 10167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:59.190467 10152 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:02:59.190541 10152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-1/wal
uuid: "40549454a29b43d1a62d6d9fbdc5a7cf"
format_stamp: "Formatted at 2026-09-29 19:02:59 on dist-test-slave-nmsz"
server_key: "7817a30a8f50c8c24007e54c07dbd6e5"
server_key_iv: "dd0483fa836555ad30e0170f6b16bb2d"
server_key_version: "encryptionkey@0"
I20260929 19:02:59.190630 10152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:59.219439 10152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:59.221589 10152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:59.221760 10152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:59.222470 10152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:02:59.222527 10152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:59.222560 10152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:02:59.222594 10152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 19:02:59 dist-test-slave-nmsz krb5kdc[9895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708579, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.66@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:02:59.230386 10152 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.66:38039
I20260929 19:02:59.230834 10152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
I20260929 19:02:59.231324 10282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.66:38039 every 8 connection(s)
I20260929 19:02:59.232916 9989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:59.222833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:52071 (local address 127.8.158.126:45077)
0929 19:02:59.222900 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:59.222903 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:59.223229 (+ 326us) server_negotiation.cc:407] Connection header received
0929 19:02:59.223998 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:59.224000 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:59.224013 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:59.224083 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:59.225164 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.225712 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.226856 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.227057 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.227162 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:59.229727 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:59.229728 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:59.229729 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:59.229737 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:59.231542 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:59.231941 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:59.231942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:59.231942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:59.232000 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:59.232270 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:59.232271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:59.232271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:59.232393 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:59.232465 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:59.232741 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 19:02:59.232818 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 19:02:59.233098 10176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:59.222835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:45077 (local address 127.8.158.66:52071)
0929 19:02:59.223056 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:59.223116 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:59.223860 (+ 744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:59.224173 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:59.224178 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:59.224630 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:59.224953 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:59.224961 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.225816 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.225817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:59.226758 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:59.226761 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.227168 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.227169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:59.227294 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:59.227793 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:02:59.227801 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:59.229635 (+ 1834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:59.231631 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:59.231637 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:59.231639 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:59.231847 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:59.232074 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:59.232075 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:59.232075 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:59.232175 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:59.232491 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:59.232497 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:59.232611 (+ 114us) client_negotiation.cc:769] Sending connection context
0929 19:02:59.232708 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 19:02:59.232853 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":79,"threads_started":1}
I20260929 19:02:59.233666 10283 heartbeater.cc:347] Connected to a master server at 127.8.158.126:45077
I20260929 19:02:59.233732 10283 heartbeater.cc:464] Registering TS with master...
I20260929 19:02:59.233906 10283 heartbeater.cc:511] Master 127.8.158.126:45077 requested a full tablet report, sending...
I20260929 19:02:59.234298 9939 ts_manager.cc:194] Registered new tserver with Master: 40549454a29b43d1a62d6d9fbdc5a7cf (127.8.158.66:38039)
I20260929 19:02:59.234699 9939 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.66@KRBTEST.COM'} at 127.8.158.66:52071
I20260929 19:02:59.240602 9989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:59.234844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:47953 (local address 127.8.158.126:45077)
0929 19:02:59.234879 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:59.234881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:59.235012 (+ 131us) server_negotiation.cc:407] Connection header received
0929 19:02:59.235495 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:59.235496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:59.235506 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:59.235551 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:59.235875 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.236377 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.237077 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.237275 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.237347 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:59.238169 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:59.238170 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:59.238170 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:59.238177 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:59.239462 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:59.239788 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:59.239789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:59.239789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:59.239849 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:59.240047 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:59.240048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:59.240049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:59.240185 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:59.240258 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:59.240443 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 19:02:59.240519 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 19:02:59.240828 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 10152
I20260929 19:02:59.240847 10289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:59.234804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:45077 (local address 127.8.158.66:47953)
0929 19:02:59.234938 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:59.234948 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:59.235414 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:59.235584 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:59.235585 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:59.235737 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:59.235785 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:59.235787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.236470 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.236471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:59.236977 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:59.236978 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.237346 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.237346 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:59.237417 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:59.237809 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:02:59.237812 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:59.238083 (+ 271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:59.239553 (+ 1470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:59.239554 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:59.239554 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:59.239716 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:59.239917 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:59.239918 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:59.239919 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:59.239975 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:59.240260 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:59.240261 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:59.240304 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 19:02:59.240419 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 19:02:59.240516 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":88,"threads_started":1}
I20260929 19:02:59.240906 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance
I20260929 19:02:59.241150 8825 external_mini_cluster.cc:1428] Setting key 523d8920a57ae2e86a2dcf662df1fccf
WARNING: no policy specified for kudu/127.8.158.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab.
Entry for principal kudu/127.8.158.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab.
I20260929 19:02:59.264856 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.67:0
--local_ip_for_outbound_sockets=127.8.158.67
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:45077
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab
--principal=kudu/127.8.158.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:42801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:02:59.346092 10292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:59.346282 10292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:59.346305 10292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:59.347460 10292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:59.347501 10292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:59.347522 10292 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 19:02:59.347538 10292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.67
I20260929 19:02:59.348968 10292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:42801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab
--principal=kudu/127.8.158.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.158.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.158.126:45077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.67
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:02:59.349191 10292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:59.349422 10292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:59.354182 10297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:59.354172 10298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:59.354285 10292 server_base.cc:1034] running on GCE node
W20260929 19:02:59.354192 10300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:59.354627 10292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:59.354833 10292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:59.356001 10292 hybrid_clock.cc:648] HybridClock initialized: now 1790708579355989 us; error 25 us; skew 500 ppm
Sep 29 19:02:59 dist-test-slave-nmsz krb5kdc[9895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708579, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:59.358424 10292 init.cc:377] Logged in from keytab as kudu/127.8.158.67@KRBTEST.COM (short username kudu)
I20260929 19:02:59.358806 10292 webserver.cc:460] Webserver started at http://127.8.158.67:39351/ using document root <none> and password file <none>
I20260929 19:02:59.358984 10292 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:59.359023 10292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:59.359107 10292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:02:59.360216 10292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-2/data/instance:
uuid: "2e543c72d124494ea1776aea0142f575"
format_stamp: "Formatted at 2026-09-29 19:02:59 on dist-test-slave-nmsz"
server_key: "a8bfb02b359092a1f6a75a688cdb2bdb"
server_key_iv: "8ac8fbb18f6dd57347c8db5f5fab61f2"
server_key_version: "encryptionkey@0"
I20260929 19:02:59.360512 10292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance:
uuid: "2e543c72d124494ea1776aea0142f575"
format_stamp: "Formatted at 2026-09-29 19:02:59 on dist-test-slave-nmsz"
server_key: "a8bfb02b359092a1f6a75a688cdb2bdb"
server_key_iv: "8ac8fbb18f6dd57347c8db5f5fab61f2"
server_key_version: "encryptionkey@0"
I20260929 19:02:59.361788 10292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 19:02:59.362490 10307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:59.362645 10292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:02:59.362739 10292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-2/wal
uuid: "2e543c72d124494ea1776aea0142f575"
format_stamp: "Formatted at 2026-09-29 19:02:59 on dist-test-slave-nmsz"
server_key: "a8bfb02b359092a1f6a75a688cdb2bdb"
server_key_iv: "8ac8fbb18f6dd57347c8db5f5fab61f2"
server_key_version: "encryptionkey@0"
I20260929 19:02:59.362805 10292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:59.383147 10292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:59.385376 10292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:59.385488 10292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:59.385944 10292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:02:59.385985 10292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:59.386008 10292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:02:59.386056 10292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:59.393008 10292 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.67:38653
I20260929 19:02:59.393069 10422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.67:38653 every 8 connection(s)
Sep 29 19:02:59 dist-test-slave-nmsz krb5kdc[9895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708579, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.67@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:02:59.393430 10292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
I20260929 19:02:59.396310 9989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:59.386504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:49249 (local address 127.8.158.126:45077)
0929 19:02:59.386576 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:59.386579 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:59.386821 (+ 242us) server_negotiation.cc:407] Connection header received
0929 19:02:59.387310 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:59.387311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:59.387325 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:59.387392 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:59.388217 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.389029 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.390609 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.390959 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.391025 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:59.393879 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:59.393880 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:59.393880 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:59.393887 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:59.395235 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:59.395593 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:59.395594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:59.395594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:59.395640 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:59.395855 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:59.395855 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:59.395855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:59.395941 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:59.395985 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:59.396188 (+ 203us) server_negotiation.cc:299] Negotiation successful
0929 19:02:59.396248 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 19:02:59.396556 10316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:59.386513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:45077 (local address 127.8.158.67:49249)
0929 19:02:59.386692 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:59.386731 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:59.387208 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:59.387479 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:59.387483 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:59.387829 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:59.388109 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:59.388116 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.389157 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.389158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:59.390512 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:59.390516 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.391054 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.391055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:59.391215 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:59.391825 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:02:59.391836 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:59.393794 (+ 1958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:59.395308 (+ 1514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:59.395313 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:59.395315 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:59.395522 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:59.395697 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:59.395697 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:59.395698 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:59.395806 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:59.395990 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:59.395994 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:59.396098 (+ 104us) client_negotiation.cc:769] Sending connection context
0929 19:02:59.396176 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 19:02:59.396304 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":112,"threads_started":1}
I20260929 19:02:59.397038 10423 heartbeater.cc:347] Connected to a master server at 127.8.158.126:45077
I20260929 19:02:59.397115 10423 heartbeater.cc:464] Registering TS with master...
I20260929 19:02:59.397307 10423 heartbeater.cc:511] Master 127.8.158.126:45077 requested a full tablet report, sending...
I20260929 19:02:59.397651 9939 ts_manager.cc:194] Registered new tserver with Master: 2e543c72d124494ea1776aea0142f575 (127.8.158.67:38653)
I20260929 19:02:59.398164 9939 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.67@KRBTEST.COM'} at 127.8.158.67:49249
I20260929 19:02:59.399881 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 10292
I20260929 19:02:59.399966 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance
I20260929 19:02:59.400144 8825 external_mini_cluster.cc:1428] Setting key 82959a011fbab88bdc8d7042a6f101f1
I20260929 19:02:59.401118 8825 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:02:59.404048 10429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:59.397972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:45077 (local address 127.8.158.67:43697)
0929 19:02:59.398094 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:59.398104 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:59.398164 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:59.398372 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:59.398373 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:59.398574 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:59.398627 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:59.398629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.399371 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.399371 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:59.400020 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:59.400023 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.400440 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.400441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:59.400524 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:59.400867 (+ 343us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:02:59.400871 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:59.401221 (+ 350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:59.403004 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:59.403005 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:59.403005 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:59.403184 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:59.403394 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:59.403395 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:59.403395 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:59.403447 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:59.403742 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:59.403743 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:59.403794 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 19:02:59.403865 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 19:02:59.403957 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":75,"threads_started":1}
I20260929 19:02:59.404125 9989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:59.398055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:43697 (local address 127.8.158.126:45077)
0929 19:02:59.398090 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:59.398092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:59.398152 (+ 60us) server_negotiation.cc:407] Connection header received
0929 19:02:59.398220 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:59.398238 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:59.398247 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:59.398301 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:59.398706 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.399248 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.400099 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.400365 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.400427 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:59.401307 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:59.401308 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:59.401309 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:59.401314 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:59.402914 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:59.403263 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:59.403264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:59.403264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:59.403328 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:59.403509 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:59.403510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:59.403510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:59.403646 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:59.403726 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:59.403914 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 19:02:59.403994 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 29 19:02:59 dist-test-slave-nmsz krb5kdc[9895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708579, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 19:02:59 dist-test-slave-nmsz krb5kdc[9895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708579, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:02:59.425611 9989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:59.418164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59452 (local address 127.8.158.126:45077)
0929 19:02:59.418227 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:59.418229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:59.418237 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:02:59.418280 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:59.418281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:59.418289 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:59.418426 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:59.418974 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.419515 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.420361 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.420548 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.420611 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:59.423070 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:59.423071 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:59.423071 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:59.423078 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:59.424430 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:59.424792 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:59.424793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:59.424793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:59.424853 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:59.425136 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:59.425137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:59.425137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:59.425244 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:59.425323 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:59.425482 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 19:02:59.425544 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 19:02:59.427484 9939 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59452:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:02:59.429060 9939 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:02:59.435801 10442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:59.431476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42282 (local address 127.8.158.66:38039)
0929 19:02:59.431901 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:59.431905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:59.431914 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:02:59.431947 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:59.431949 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:59.431962 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:59.432043 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:59.432229 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.433135 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.434649 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.435461 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.435527 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:59.435581 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:59.435670 (+ 89us) server_negotiation.cc:299] Negotiation successful
0929 19:02:59.435707 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":62,"threads_started":1}
I20260929 19:02:59.435830 10440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:59.431370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.66:38039 (local address 127.0.0.1:42282)
0929 19:02:59.431479 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:59.431517 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:59.431575 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:59.432051 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:59.432052 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:59.432054 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:59.432124 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:59.432126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.433222 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.433223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:59.434543 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:59.434546 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.435530 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.435531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:59.435590 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:59.435594 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:02:59.435656 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 19:02:59.435696 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":11,"thread_start_us":66,"threads_started":1}
I20260929 19:02:59.435904 10443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:59.431544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43994 (local address 127.8.158.67:38653)
0929 19:02:59.431673 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:59.431678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:59.431772 (+ 94us) server_negotiation.cc:407] Connection header received
0929 19:02:59.431851 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:59.431853 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:59.431870 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:59.431950 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:59.432200 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.433437 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.434681 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.435347 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.435415 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:59.435468 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:59.435761 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 19:02:59.435801 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":67,"threads_started":1}
I20260929 19:02:59.436056 10441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:59.431372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.67:38653 (local address 127.0.0.1:43994)
0929 19:02:59.431693 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:59.431702 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:59.431765 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:59.432033 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:59.432041 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:59.432043 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:59.432105 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:59.432107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.433540 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.433541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:59.434601 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:59.434603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.435442 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.435443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:59.435527 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:59.435541 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 19:02:59.435606 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 19:02:59.435664 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":85,"threads_started":1}
I20260929 19:02:59.436542 10444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:59.431575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:45603 (local address 127.0.0.1:48128)
0929 19:02:59.431823 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:59.431834 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:59.431883 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:59.432360 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:59.432360 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:59.432362 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:59.432394 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:59.432396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.433816 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.433817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:59.434951 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:59.434954 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.436335 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.436336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:59.436392 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:59.436396 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:02:59.436443 (+ 47us) client_negotiation.cc:240] Negotiation successful
0929 19:02:59.436479 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":1,"thread_start_us":83,"threads_started":1}
I20260929 19:02:59.436587 10217 tablet_service.cc:1467] Processing CreateTablet for tablet 6484fef9b1844f94a0d1d03164b8b729 (DEFAULT_TABLE table=test-table [id=3e6ed86744464e5d91f9080fe8903228]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:02:59.436729 10357 tablet_service.cc:1467] Processing CreateTablet for tablet 6484fef9b1844f94a0d1d03164b8b729 (DEFAULT_TABLE table=test-table [id=3e6ed86744464e5d91f9080fe8903228]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:02:59.436932 10445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:59.431819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48128 (local address 127.8.158.65:45603)
0929 19:02:59.432001 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:59.432008 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:59.432019 (+ 11us) server_negotiation.cc:407] Connection header received
0929 19:02:59.432181 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:59.432197 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:59.432224 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:59.432299 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:59.432478 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.433742 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.435035 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.435772 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.435832 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:59.436715 (+ 883us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:59.436742 (+ 27us) server_negotiation.cc:299] Negotiation successful
0929 19:02:59.436785 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":82,"threads_started":1}
I20260929 19:02:59.437001 10217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6484fef9b1844f94a0d1d03164b8b729. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:59.437062 10357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6484fef9b1844f94a0d1d03164b8b729. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:59.437688 10077 tablet_service.cc:1467] Processing CreateTablet for tablet 6484fef9b1844f94a0d1d03164b8b729 (DEFAULT_TABLE table=test-table [id=3e6ed86744464e5d91f9080fe8903228]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:02:59.437985 10077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6484fef9b1844f94a0d1d03164b8b729. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:59.439486 10446 tablet_bootstrap.cc:492] T 6484fef9b1844f94a0d1d03164b8b729 P 2e543c72d124494ea1776aea0142f575: Bootstrap starting.
I20260929 19:02:59.440363 10446 tablet_bootstrap.cc:654] T 6484fef9b1844f94a0d1d03164b8b729 P 2e543c72d124494ea1776aea0142f575: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:59.440665 10446 log.cc:824] T 6484fef9b1844f94a0d1d03164b8b729 P 2e543c72d124494ea1776aea0142f575: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:59.440733 10447 tablet_bootstrap.cc:492] T 6484fef9b1844f94a0d1d03164b8b729 P 40549454a29b43d1a62d6d9fbdc5a7cf: Bootstrap starting.
I20260929 19:02:59.441283 10447 tablet_bootstrap.cc:654] T 6484fef9b1844f94a0d1d03164b8b729 P 40549454a29b43d1a62d6d9fbdc5a7cf: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:59.441453 10446 tablet_bootstrap.cc:492] T 6484fef9b1844f94a0d1d03164b8b729 P 2e543c72d124494ea1776aea0142f575: No bootstrap required, opened a new log
I20260929 19:02:59.441479 10449 tablet_bootstrap.cc:492] T 6484fef9b1844f94a0d1d03164b8b729 P bfeeef88c11247529376decbf76640f4: Bootstrap starting.
I20260929 19:02:59.441509 10446 ts_tablet_manager.cc:1397] T 6484fef9b1844f94a0d1d03164b8b729 P 2e543c72d124494ea1776aea0142f575: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:02:59.441592 10447 log.cc:824] T 6484fef9b1844f94a0d1d03164b8b729 P 40549454a29b43d1a62d6d9fbdc5a7cf: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:59.442107 10449 tablet_bootstrap.cc:654] T 6484fef9b1844f94a0d1d03164b8b729 P bfeeef88c11247529376decbf76640f4: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:59.442484 10449 log.cc:824] T 6484fef9b1844f94a0d1d03164b8b729 P bfeeef88c11247529376decbf76640f4: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:59.442706 10447 tablet_bootstrap.cc:492] T 6484fef9b1844f94a0d1d03164b8b729 P 40549454a29b43d1a62d6d9fbdc5a7cf: No bootstrap required, opened a new log
I20260929 19:02:59.442780 10447 ts_tablet_manager.cc:1397] T 6484fef9b1844f94a0d1d03164b8b729 P 40549454a29b43d1a62d6d9fbdc5a7cf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:02:59.443109 10449 tablet_bootstrap.cc:492] T 6484fef9b1844f94a0d1d03164b8b729 P bfeeef88c11247529376decbf76640f4: No bootstrap required, opened a new log
I20260929 19:02:59.443190 10449 ts_tablet_manager.cc:1397] T 6484fef9b1844f94a0d1d03164b8b729 P bfeeef88c11247529376decbf76640f4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 19:02:59.443646 10446 raft_consensus.cc:348] T 6484fef9b1844f94a0d1d03164b8b729 P 2e543c72d124494ea1776aea0142f575 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e543c72d124494ea1776aea0142f575" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38653 } } peers { permanent_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38039 } } peers { permanent_uuid: "bfeeef88c11247529376decbf76640f4" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 45603 } }
I20260929 19:02:59.443816 10446 raft_consensus.cc:374] T 6484fef9b1844f94a0d1d03164b8b729 P 2e543c72d124494ea1776aea0142f575 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:59.443854 10446 raft_consensus.cc:729] T 6484fef9b1844f94a0d1d03164b8b729 P 2e543c72d124494ea1776aea0142f575 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e543c72d124494ea1776aea0142f575, State: Initialized, Role: FOLLOWER
I20260929 19:02:59.443985 10446 consensus_queue.cc:260] T 6484fef9b1844f94a0d1d03164b8b729 P 2e543c72d124494ea1776aea0142f575 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e543c72d124494ea1776aea0142f575" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38653 } } peers { permanent_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38039 } } peers { permanent_uuid: "bfeeef88c11247529376decbf76640f4" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 45603 } }
I20260929 19:02:59.444242 10446 ts_tablet_manager.cc:1428] T 6484fef9b1844f94a0d1d03164b8b729 P 2e543c72d124494ea1776aea0142f575: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 19:02:59.444367 10423 heartbeater.cc:503] Master 127.8.158.126:45077 was elected leader, sending a full tablet report...
I20260929 19:02:59.444655 10447 raft_consensus.cc:348] T 6484fef9b1844f94a0d1d03164b8b729 P 40549454a29b43d1a62d6d9fbdc5a7cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e543c72d124494ea1776aea0142f575" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38653 } } peers { permanent_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38039 } } peers { permanent_uuid: "bfeeef88c11247529376decbf76640f4" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 45603 } }
I20260929 19:02:59.444762 10447 raft_consensus.cc:374] T 6484fef9b1844f94a0d1d03164b8b729 P 40549454a29b43d1a62d6d9fbdc5a7cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:59.444782 10447 raft_consensus.cc:729] T 6484fef9b1844f94a0d1d03164b8b729 P 40549454a29b43d1a62d6d9fbdc5a7cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40549454a29b43d1a62d6d9fbdc5a7cf, State: Initialized, Role: FOLLOWER
I20260929 19:02:59.444883 10447 consensus_queue.cc:260] T 6484fef9b1844f94a0d1d03164b8b729 P 40549454a29b43d1a62d6d9fbdc5a7cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e543c72d124494ea1776aea0142f575" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38653 } } peers { permanent_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38039 } } peers { permanent_uuid: "bfeeef88c11247529376decbf76640f4" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 45603 } }
I20260929 19:02:59.444906 10449 raft_consensus.cc:348] T 6484fef9b1844f94a0d1d03164b8b729 P bfeeef88c11247529376decbf76640f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e543c72d124494ea1776aea0142f575" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38653 } } peers { permanent_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38039 } } peers { permanent_uuid: "bfeeef88c11247529376decbf76640f4" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 45603 } }
I20260929 19:02:59.445060 10449 raft_consensus.cc:374] T 6484fef9b1844f94a0d1d03164b8b729 P bfeeef88c11247529376decbf76640f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:59.445101 10449 raft_consensus.cc:729] T 6484fef9b1844f94a0d1d03164b8b729 P bfeeef88c11247529376decbf76640f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfeeef88c11247529376decbf76640f4, State: Initialized, Role: FOLLOWER
I20260929 19:02:59.445125 10447 ts_tablet_manager.cc:1428] T 6484fef9b1844f94a0d1d03164b8b729 P 40549454a29b43d1a62d6d9fbdc5a7cf: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 19:02:59.445134 10283 heartbeater.cc:503] Master 127.8.158.126:45077 was elected leader, sending a full tablet report...
I20260929 19:02:59.445192 10449 consensus_queue.cc:260] T 6484fef9b1844f94a0d1d03164b8b729 P bfeeef88c11247529376decbf76640f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e543c72d124494ea1776aea0142f575" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38653 } } peers { permanent_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38039 } } peers { permanent_uuid: "bfeeef88c11247529376decbf76640f4" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 45603 } }
I20260929 19:02:59.445446 10449 ts_tablet_manager.cc:1428] T 6484fef9b1844f94a0d1d03164b8b729 P bfeeef88c11247529376decbf76640f4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 19:02:59.445473 10143 heartbeater.cc:503] Master 127.8.158.126:45077 was elected leader, sending a full tablet report...
W20260929 19:02:59.481503 10284 tablet.cc:2369] T 6484fef9b1844f94a0d1d03164b8b729 P 40549454a29b43d1a62d6d9fbdc5a7cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:02:59.495488 10452 raft_consensus.cc:482] T 6484fef9b1844f94a0d1d03164b8b729 P 2e543c72d124494ea1776aea0142f575 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:02:59.495594 10452 raft_consensus.cc:504] T 6484fef9b1844f94a0d1d03164b8b729 P 2e543c72d124494ea1776aea0142f575 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e543c72d124494ea1776aea0142f575" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38653 } } peers { permanent_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38039 } } peers { permanent_uuid: "bfeeef88c11247529376decbf76640f4" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 45603 } }
I20260929 19:02:59.495872 10452 leader_election.cc:288] T 6484fef9b1844f94a0d1d03164b8b729 P 2e543c72d124494ea1776aea0142f575 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40549454a29b43d1a62d6d9fbdc5a7cf (127.8.158.66:38039), bfeeef88c11247529376decbf76640f4 (127.8.158.65:45603)
I20260929 19:02:59.498945 10316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:59.496070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.66:38039 (local address 127.8.158.67:33151)
0929 19:02:59.496134 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:59.496149 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:59.496213 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:59.496383 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:59.496384 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:59.496386 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:59.496449 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:59.496450 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.497222 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.497223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:59.498104 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:59.498106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.498679 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.498680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:59.498766 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:59.498769 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:02:59.498835 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 19:02:59.498873 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260929 19:02:59.498945 10442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:59.496155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:33151 (local address 127.8.158.66:38039)
0929 19:02:59.496218 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:59.496220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:59.496230 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:02:59.496276 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:59.496277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:59.496286 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:59.496342 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:59.496519 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.497142 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.498192 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.498588 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.498673 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:59.498681 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:59.498849 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 19:02:59.498888 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 19:02:59.499238 10237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6484fef9b1844f94a0d1d03164b8b729" candidate_uuid: "2e543c72d124494ea1776aea0142f575" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf" is_pre_election: true
I20260929 19:02:59.499385 10237 raft_consensus.cc:2454] T 6484fef9b1844f94a0d1d03164b8b729 P 40549454a29b43d1a62d6d9fbdc5a7cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e543c72d124494ea1776aea0142f575 in term 0.
I20260929 19:02:59.499441 10455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:59.496198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:45603 (local address 127.8.158.67:42389)
0929 19:02:59.496377 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:59.496388 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:59.496480 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:59.496636 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:59.496637 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:59.496639 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:59.496739 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:59.496741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.497530 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.497531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:59.498480 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:59.498482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.499142 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.499144 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:59.499231 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:59.499234 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:02:59.499305 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 19:02:59.499344 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":106,"threads_started":1}
I20260929 19:02:59.499428 10445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:59.496296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:42389 (local address 127.8.158.65:45603)
0929 19:02:59.496347 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:59.496349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:59.496466 (+ 117us) server_negotiation.cc:407] Connection header received
0929 19:02:59.496542 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:59.496544 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:59.496554 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:59.496631 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:59.496834 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.497433 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.498571 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.498973 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.499069 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:59.499079 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:59.499309 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 19:02:59.499345 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 19:02:59.499756 10311 leader_election.cc:302] T 6484fef9b1844f94a0d1d03164b8b729 P 2e543c72d124494ea1776aea0142f575 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e543c72d124494ea1776aea0142f575, 40549454a29b43d1a62d6d9fbdc5a7cf; no voters:
I20260929 19:02:59.499889 10452 raft_consensus.cc:2781] T 6484fef9b1844f94a0d1d03164b8b729 P 2e543c72d124494ea1776aea0142f575 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:02:59.499953 10452 raft_consensus.cc:482] T 6484fef9b1844f94a0d1d03164b8b729 P 2e543c72d124494ea1776aea0142f575 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:02:59.499989 10452 raft_consensus.cc:3037] T 6484fef9b1844f94a0d1d03164b8b729 P 2e543c72d124494ea1776aea0142f575 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:59.500098 10097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6484fef9b1844f94a0d1d03164b8b729" candidate_uuid: "2e543c72d124494ea1776aea0142f575" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfeeef88c11247529376decbf76640f4" is_pre_election: true
I20260929 19:02:59.500208 10097 raft_consensus.cc:2454] T 6484fef9b1844f94a0d1d03164b8b729 P bfeeef88c11247529376decbf76640f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e543c72d124494ea1776aea0142f575 in term 0.
I20260929 19:02:59.500762 10452 raft_consensus.cc:504] T 6484fef9b1844f94a0d1d03164b8b729 P 2e543c72d124494ea1776aea0142f575 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e543c72d124494ea1776aea0142f575" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38653 } } peers { permanent_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38039 } } peers { permanent_uuid: "bfeeef88c11247529376decbf76640f4" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 45603 } }
I20260929 19:02:59.500895 10452 leader_election.cc:288] T 6484fef9b1844f94a0d1d03164b8b729 P 2e543c72d124494ea1776aea0142f575 [CANDIDATE]: Term 1 election: Requested vote from peers 40549454a29b43d1a62d6d9fbdc5a7cf (127.8.158.66:38039), bfeeef88c11247529376decbf76640f4 (127.8.158.65:45603)
I20260929 19:02:59.501097 10237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6484fef9b1844f94a0d1d03164b8b729" candidate_uuid: "2e543c72d124494ea1776aea0142f575" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf"
I20260929 19:02:59.501122 10097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6484fef9b1844f94a0d1d03164b8b729" candidate_uuid: "2e543c72d124494ea1776aea0142f575" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfeeef88c11247529376decbf76640f4"
I20260929 19:02:59.501175 10237 raft_consensus.cc:3037] T 6484fef9b1844f94a0d1d03164b8b729 P 40549454a29b43d1a62d6d9fbdc5a7cf [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:59.501180 10097 raft_consensus.cc:3037] T 6484fef9b1844f94a0d1d03164b8b729 P bfeeef88c11247529376decbf76640f4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:59.501849 10237 raft_consensus.cc:2454] T 6484fef9b1844f94a0d1d03164b8b729 P 40549454a29b43d1a62d6d9fbdc5a7cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e543c72d124494ea1776aea0142f575 in term 1.
I20260929 19:02:59.501961 10097 raft_consensus.cc:2454] T 6484fef9b1844f94a0d1d03164b8b729 P bfeeef88c11247529376decbf76640f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e543c72d124494ea1776aea0142f575 in term 1.
I20260929 19:02:59.502076 10311 leader_election.cc:302] T 6484fef9b1844f94a0d1d03164b8b729 P 2e543c72d124494ea1776aea0142f575 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e543c72d124494ea1776aea0142f575, 40549454a29b43d1a62d6d9fbdc5a7cf; no voters:
I20260929 19:02:59.502185 10452 raft_consensus.cc:2781] T 6484fef9b1844f94a0d1d03164b8b729 P 2e543c72d124494ea1776aea0142f575 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:02:59.502440 10452 raft_consensus.cc:686] T 6484fef9b1844f94a0d1d03164b8b729 P 2e543c72d124494ea1776aea0142f575 [term 1 LEADER]: Becoming Leader. State: Replica: 2e543c72d124494ea1776aea0142f575, State: Running, Role: LEADER
I20260929 19:02:59.502562 10452 consensus_queue.cc:237] T 6484fef9b1844f94a0d1d03164b8b729 P 2e543c72d124494ea1776aea0142f575 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e543c72d124494ea1776aea0142f575" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38653 } } peers { permanent_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38039 } } peers { permanent_uuid: "bfeeef88c11247529376decbf76640f4" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 45603 } }
I20260929 19:02:59.503425 9940 catalog_manager.cc:5495] T 6484fef9b1844f94a0d1d03164b8b729 P 2e543c72d124494ea1776aea0142f575 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e543c72d124494ea1776aea0142f575 (127.8.158.67). New cstate: current_term: 1 leader_uuid: "2e543c72d124494ea1776aea0142f575" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e543c72d124494ea1776aea0142f575" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfeeef88c11247529376decbf76640f4" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 45603 } health_report { overall_health: UNKNOWN } } }
I20260929 19:02:59.512331 10443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:59.509878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43996 (local address 127.8.158.67:38653)
0929 19:02:59.509924 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:59.509927 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:59.509935 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:02:59.510005 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:59.510006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:59.510023 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:59.510097 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:02:59.510246 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.510735 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.511527 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.511715 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.511796 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:59.511999 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:02:59.512091 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:02:59.512169 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:59.512241 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 19:02:59.512259 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 19:02:59.514093 10097 raft_consensus.cc:1261] T 6484fef9b1844f94a0d1d03164b8b729 P bfeeef88c11247529376decbf76640f4 [term 1 FOLLOWER]: Refusing update from remote peer 2e543c72d124494ea1776aea0142f575: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:02:59.514129 10237 raft_consensus.cc:1261] T 6484fef9b1844f94a0d1d03164b8b729 P 40549454a29b43d1a62d6d9fbdc5a7cf [term 1 FOLLOWER]: Refusing update from remote peer 2e543c72d124494ea1776aea0142f575: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:02:59.514461 10452 consensus_queue.cc:1035] T 6484fef9b1844f94a0d1d03164b8b729 P 2e543c72d124494ea1776aea0142f575 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:02:59.514591 10456 consensus_queue.cc:1035] T 6484fef9b1844f94a0d1d03164b8b729 P 2e543c72d124494ea1776aea0142f575 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfeeef88c11247529376decbf76640f4" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 45603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:02:59.520288 9940 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:02:59.520380 9940 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:02:59.521143 9940 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 869266291b584c3583320a409b4e2c9c: Sending DeleteTablet for 3 replicas of tablet 6484fef9b1844f94a0d1d03164b8b729
I20260929 19:02:59.521507 10357 tablet_service.cc:1514] Processing DeleteTablet for tablet 6484fef9b1844f94a0d1d03164b8b729 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:02:59 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:43994
I20260929 19:02:59.521610 10217 tablet_service.cc:1514] Processing DeleteTablet for tablet 6484fef9b1844f94a0d1d03164b8b729 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:02:59 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:42282
I20260929 19:02:59.521725 10077 tablet_service.cc:1514] Processing DeleteTablet for tablet 6484fef9b1844f94a0d1d03164b8b729 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:02:59 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:48128
I20260929 19:02:59.521799 9940 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59452:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:02:59.521938 9940 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:02:59.521872 10469 tablet_replica.cc:330] T 6484fef9b1844f94a0d1d03164b8b729 P 40549454a29b43d1a62d6d9fbdc5a7cf: stopping tablet replica
I20260929 19:02:59.522099 10470 tablet_replica.cc:330] T 6484fef9b1844f94a0d1d03164b8b729 P bfeeef88c11247529376decbf76640f4: stopping tablet replica
I20260929 19:02:59.522109 10469 raft_consensus.cc:2229] T 6484fef9b1844f94a0d1d03164b8b729 P 40549454a29b43d1a62d6d9fbdc5a7cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:02:59.522240 10470 raft_consensus.cc:2229] T 6484fef9b1844f94a0d1d03164b8b729 P bfeeef88c11247529376decbf76640f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:02:59.522271 10469 raft_consensus.cc:2258] T 6484fef9b1844f94a0d1d03164b8b729 P 40549454a29b43d1a62d6d9fbdc5a7cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:02:59.522723 10470 raft_consensus.cc:2258] T 6484fef9b1844f94a0d1d03164b8b729 P bfeeef88c11247529376decbf76640f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:02:59.523236 10357 tablet_service.cc:1467] Processing CreateTablet for tablet e0f451a7220d414084f8c5515de8c9d5 (DEFAULT_TABLE table=test-table [id=6ce7c1fd4f77460f80805196fd09954f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:02:59.523377 10357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0f451a7220d414084f8c5515de8c9d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:59.523571 10217 tablet_service.cc:1467] Processing CreateTablet for tablet e0f451a7220d414084f8c5515de8c9d5 (DEFAULT_TABLE table=test-table [id=6ce7c1fd4f77460f80805196fd09954f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:02:59.523663 10217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0f451a7220d414084f8c5515de8c9d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:59.524845 10446 tablet_bootstrap.cc:492] T e0f451a7220d414084f8c5515de8c9d5 P 2e543c72d124494ea1776aea0142f575: Bootstrap starting.
I20260929 19:02:59.524883 10077 tablet_service.cc:1467] Processing CreateTablet for tablet e0f451a7220d414084f8c5515de8c9d5 (DEFAULT_TABLE table=test-table [id=6ce7c1fd4f77460f80805196fd09954f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:02:59.524986 10077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0f451a7220d414084f8c5515de8c9d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:59.525233 10446 tablet_bootstrap.cc:654] T e0f451a7220d414084f8c5515de8c9d5 P 2e543c72d124494ea1776aea0142f575: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:59.525583 10447 tablet_bootstrap.cc:492] T e0f451a7220d414084f8c5515de8c9d5 P 40549454a29b43d1a62d6d9fbdc5a7cf: Bootstrap starting.
I20260929 19:02:59.525987 10447 tablet_bootstrap.cc:654] T e0f451a7220d414084f8c5515de8c9d5 P 40549454a29b43d1a62d6d9fbdc5a7cf: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:59.526548 10446 tablet_bootstrap.cc:492] T e0f451a7220d414084f8c5515de8c9d5 P 2e543c72d124494ea1776aea0142f575: No bootstrap required, opened a new log
I20260929 19:02:59.526615 10446 ts_tablet_manager.cc:1397] T e0f451a7220d414084f8c5515de8c9d5 P 2e543c72d124494ea1776aea0142f575: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:02:59.526635 10449 tablet_bootstrap.cc:492] T e0f451a7220d414084f8c5515de8c9d5 P bfeeef88c11247529376decbf76640f4: Bootstrap starting.
I20260929 19:02:59.526746 10446 raft_consensus.cc:348] T e0f451a7220d414084f8c5515de8c9d5 P 2e543c72d124494ea1776aea0142f575 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfeeef88c11247529376decbf76640f4" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 45603 } } peers { permanent_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38039 } } peers { permanent_uuid: "2e543c72d124494ea1776aea0142f575" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38653 } }
I20260929 19:02:59.526809 10446 raft_consensus.cc:374] T e0f451a7220d414084f8c5515de8c9d5 P 2e543c72d124494ea1776aea0142f575 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:59.526834 10446 raft_consensus.cc:729] T e0f451a7220d414084f8c5515de8c9d5 P 2e543c72d124494ea1776aea0142f575 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e543c72d124494ea1776aea0142f575, State: Initialized, Role: FOLLOWER
I20260929 19:02:59.526865 10446 consensus_queue.cc:260] T e0f451a7220d414084f8c5515de8c9d5 P 2e543c72d124494ea1776aea0142f575 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfeeef88c11247529376decbf76640f4" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 45603 } } peers { permanent_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38039 } } peers { permanent_uuid: "2e543c72d124494ea1776aea0142f575" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38653 } }
I20260929 19:02:59.526944 10446 ts_tablet_manager.cc:1428] T e0f451a7220d414084f8c5515de8c9d5 P 2e543c72d124494ea1776aea0142f575: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:59.527109 10449 tablet_bootstrap.cc:654] T e0f451a7220d414084f8c5515de8c9d5 P bfeeef88c11247529376decbf76640f4: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:59.527309 10470 ts_tablet_manager.cc:1905] T 6484fef9b1844f94a0d1d03164b8b729 P bfeeef88c11247529376decbf76640f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:02:59.527694 10469 ts_tablet_manager.cc:1905] T 6484fef9b1844f94a0d1d03164b8b729 P 40549454a29b43d1a62d6d9fbdc5a7cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:02:59.527832 10447 tablet_bootstrap.cc:492] T e0f451a7220d414084f8c5515de8c9d5 P 40549454a29b43d1a62d6d9fbdc5a7cf: No bootstrap required, opened a new log
I20260929 19:02:59.527887 10447 ts_tablet_manager.cc:1397] T e0f451a7220d414084f8c5515de8c9d5 P 40549454a29b43d1a62d6d9fbdc5a7cf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:02:59.528033 10447 raft_consensus.cc:348] T e0f451a7220d414084f8c5515de8c9d5 P 40549454a29b43d1a62d6d9fbdc5a7cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfeeef88c11247529376decbf76640f4" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 45603 } } peers { permanent_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38039 } } peers { permanent_uuid: "2e543c72d124494ea1776aea0142f575" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38653 } }
I20260929 19:02:59.528102 10447 raft_consensus.cc:374] T e0f451a7220d414084f8c5515de8c9d5 P 40549454a29b43d1a62d6d9fbdc5a7cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:59.528124 10447 raft_consensus.cc:729] T e0f451a7220d414084f8c5515de8c9d5 P 40549454a29b43d1a62d6d9fbdc5a7cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40549454a29b43d1a62d6d9fbdc5a7cf, State: Initialized, Role: FOLLOWER
I20260929 19:02:59.528169 10447 consensus_queue.cc:260] T e0f451a7220d414084f8c5515de8c9d5 P 40549454a29b43d1a62d6d9fbdc5a7cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfeeef88c11247529376decbf76640f4" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 45603 } } peers { permanent_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38039 } } peers { permanent_uuid: "2e543c72d124494ea1776aea0142f575" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38653 } }
I20260929 19:02:59.528259 10447 ts_tablet_manager.cc:1428] T e0f451a7220d414084f8c5515de8c9d5 P 40549454a29b43d1a62d6d9fbdc5a7cf: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 19:02:59.529264 10469 ts_tablet_manager.cc:1918] T 6484fef9b1844f94a0d1d03164b8b729 P 40549454a29b43d1a62d6d9fbdc5a7cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:02:59.529386 10469 log.cc:1196] T 6484fef9b1844f94a0d1d03164b8b729 P 40549454a29b43d1a62d6d9fbdc5a7cf: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-1/wal/wals/6484fef9b1844f94a0d1d03164b8b729
I20260929 19:02:59.530064 10469 ts_tablet_manager.cc:1939] T 6484fef9b1844f94a0d1d03164b8b729 P 40549454a29b43d1a62d6d9fbdc5a7cf: Deleting consensus metadata
I20260929 19:02:59.530424 10468 tablet_replica.cc:330] T 6484fef9b1844f94a0d1d03164b8b729 P 2e543c72d124494ea1776aea0142f575: stopping tablet replica
I20260929 19:02:59.530530 10468 raft_consensus.cc:2229] T 6484fef9b1844f94a0d1d03164b8b729 P 2e543c72d124494ea1776aea0142f575 [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:02:59.530589 9927 catalog_manager.cc:4841] TS 40549454a29b43d1a62d6d9fbdc5a7cf (127.8.158.66:38039): tablet 6484fef9b1844f94a0d1d03164b8b729 (table test-table [id=3e6ed86744464e5d91f9080fe8903228]) successfully deleted
I20260929 19:02:59.530629 10468 raft_consensus.cc:2258] T 6484fef9b1844f94a0d1d03164b8b729 P 2e543c72d124494ea1776aea0142f575 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:02:59.530704 10470 ts_tablet_manager.cc:1918] T 6484fef9b1844f94a0d1d03164b8b729 P bfeeef88c11247529376decbf76640f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:02:59.530777 10470 log.cc:1196] T 6484fef9b1844f94a0d1d03164b8b729 P bfeeef88c11247529376decbf76640f4: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-0/wal/wals/6484fef9b1844f94a0d1d03164b8b729
I20260929 19:02:59.530824 10449 tablet_bootstrap.cc:492] T e0f451a7220d414084f8c5515de8c9d5 P bfeeef88c11247529376decbf76640f4: No bootstrap required, opened a new log
I20260929 19:02:59.530876 10449 ts_tablet_manager.cc:1397] T e0f451a7220d414084f8c5515de8c9d5 P bfeeef88c11247529376decbf76640f4: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260929 19:02:59.531011 10468 ts_tablet_manager.cc:1905] T 6484fef9b1844f94a0d1d03164b8b729 P 2e543c72d124494ea1776aea0142f575: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:02:59.531023 10449 raft_consensus.cc:348] T e0f451a7220d414084f8c5515de8c9d5 P bfeeef88c11247529376decbf76640f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfeeef88c11247529376decbf76640f4" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 45603 } } peers { permanent_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38039 } } peers { permanent_uuid: "2e543c72d124494ea1776aea0142f575" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38653 } }
I20260929 19:02:59.531049 10470 ts_tablet_manager.cc:1939] T 6484fef9b1844f94a0d1d03164b8b729 P bfeeef88c11247529376decbf76640f4: Deleting consensus metadata
I20260929 19:02:59.531083 10449 raft_consensus.cc:374] T e0f451a7220d414084f8c5515de8c9d5 P bfeeef88c11247529376decbf76640f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:59.531137 10449 raft_consensus.cc:729] T e0f451a7220d414084f8c5515de8c9d5 P bfeeef88c11247529376decbf76640f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfeeef88c11247529376decbf76640f4, State: Initialized, Role: FOLLOWER
I20260929 19:02:59.531181 10449 consensus_queue.cc:260] T e0f451a7220d414084f8c5515de8c9d5 P bfeeef88c11247529376decbf76640f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfeeef88c11247529376decbf76640f4" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 45603 } } peers { permanent_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38039 } } peers { permanent_uuid: "2e543c72d124494ea1776aea0142f575" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38653 } }
I20260929 19:02:59.531275 10449 ts_tablet_manager.cc:1428] T e0f451a7220d414084f8c5515de8c9d5 P bfeeef88c11247529376decbf76640f4: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260929 19:02:59.531495 9927 catalog_manager.cc:4841] TS bfeeef88c11247529376decbf76640f4 (127.8.158.65:45603): tablet 6484fef9b1844f94a0d1d03164b8b729 (table test-table [id=3e6ed86744464e5d91f9080fe8903228]) successfully deleted
I20260929 19:02:59.532279 10468 ts_tablet_manager.cc:1918] T 6484fef9b1844f94a0d1d03164b8b729 P 2e543c72d124494ea1776aea0142f575: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:02:59.532331 10468 log.cc:1196] T 6484fef9b1844f94a0d1d03164b8b729 P 2e543c72d124494ea1776aea0142f575: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-2/wal/wals/6484fef9b1844f94a0d1d03164b8b729
I20260929 19:02:59.532579 10468 ts_tablet_manager.cc:1939] T 6484fef9b1844f94a0d1d03164b8b729 P 2e543c72d124494ea1776aea0142f575: Deleting consensus metadata
I20260929 19:02:59.532883 9925 catalog_manager.cc:4841] TS 2e543c72d124494ea1776aea0142f575 (127.8.158.67:38653): tablet 6484fef9b1844f94a0d1d03164b8b729 (table test-table [id=3e6ed86744464e5d91f9080fe8903228]) successfully deleted
I20260929 19:02:59.544852 10454 raft_consensus.cc:482] T e0f451a7220d414084f8c5515de8c9d5 P bfeeef88c11247529376decbf76640f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:02:59.544929 10454 raft_consensus.cc:504] T e0f451a7220d414084f8c5515de8c9d5 P bfeeef88c11247529376decbf76640f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfeeef88c11247529376decbf76640f4" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 45603 } } peers { permanent_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38039 } } peers { permanent_uuid: "2e543c72d124494ea1776aea0142f575" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38653 } }
I20260929 19:02:59.545204 10454 leader_election.cc:288] T e0f451a7220d414084f8c5515de8c9d5 P bfeeef88c11247529376decbf76640f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40549454a29b43d1a62d6d9fbdc5a7cf (127.8.158.66:38039), 2e543c72d124494ea1776aea0142f575 (127.8.158.67:38653)
I20260929 19:02:59.548686 10036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:59.545353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.66:38039 (local address 127.8.158.65:35619)
0929 19:02:59.545404 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:59.545416 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:59.545456 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:59.545682 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:59.545683 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:59.545685 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:59.545755 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:59.545769 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.546706 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.546707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:59.547827 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:59.547830 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.548365 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.548367 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:59.548450 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:59.548453 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:02:59.548543 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 19:02:59.548584 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":8}
I20260929 19:02:59.548710 10442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:59.545459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:35619 (local address 127.8.158.66:38039)
0929 19:02:59.545499 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:59.545501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:59.545512 (+ 11us) server_negotiation.cc:407] Connection header received
0929 19:02:59.545535 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:59.545536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:59.545544 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:59.545603 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:59.545842 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.546619 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.547949 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.548285 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.548370 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:59.548378 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:59.548603 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 19:02:59.548647 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 19:02:59.548827 10443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:59.545658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:47277 (local address 127.8.158.67:38653)
0929 19:02:59.545695 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:59.545697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:59.545703 (+ 6us) server_negotiation.cc:407] Connection header received
0929 19:02:59.545723 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:59.545724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:59.545742 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:59.545789 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:59.545945 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.546478 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.547918 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.548481 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.548552 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:59.548559 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:59.548698 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 19:02:59.548740 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 19:02:59.549024 10474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:59.545353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.67:38653 (local address 127.8.158.65:47277)
0929 19:02:59.545479 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:59.545487 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:59.545527 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:59.545809 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:59.545810 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:59.545811 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:59.545874 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:59.545875 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.546556 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.546557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:59.547827 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:59.547830 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.548551 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.548552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:59.548613 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:59.548616 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:02:59.548902 (+ 286us) client_negotiation.cc:240] Negotiation successful
0929 19:02:59.548945 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":11,"thread_start_us":48,"threads_started":1}
I20260929 19:02:59.549088 10237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f451a7220d414084f8c5515de8c9d5" candidate_uuid: "bfeeef88c11247529376decbf76640f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf" is_pre_election: true
I20260929 19:02:59.549166 10237 raft_consensus.cc:2454] T e0f451a7220d414084f8c5515de8c9d5 P 40549454a29b43d1a62d6d9fbdc5a7cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfeeef88c11247529376decbf76640f4 in term 0.
I20260929 19:02:59.549260 10377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f451a7220d414084f8c5515de8c9d5" candidate_uuid: "bfeeef88c11247529376decbf76640f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e543c72d124494ea1776aea0142f575" is_pre_election: true
I20260929 19:02:59.549320 10031 leader_election.cc:302] T e0f451a7220d414084f8c5515de8c9d5 P bfeeef88c11247529376decbf76640f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40549454a29b43d1a62d6d9fbdc5a7cf, bfeeef88c11247529376decbf76640f4; no voters:
I20260929 19:02:59.549387 10377 raft_consensus.cc:2454] T e0f451a7220d414084f8c5515de8c9d5 P 2e543c72d124494ea1776aea0142f575 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfeeef88c11247529376decbf76640f4 in term 0.
I20260929 19:02:59.549436 10454 raft_consensus.cc:2781] T e0f451a7220d414084f8c5515de8c9d5 P bfeeef88c11247529376decbf76640f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:02:59.549475 10454 raft_consensus.cc:482] T e0f451a7220d414084f8c5515de8c9d5 P bfeeef88c11247529376decbf76640f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:02:59.549499 10454 raft_consensus.cc:3037] T e0f451a7220d414084f8c5515de8c9d5 P bfeeef88c11247529376decbf76640f4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:59.550244 10454 raft_consensus.cc:504] T e0f451a7220d414084f8c5515de8c9d5 P bfeeef88c11247529376decbf76640f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfeeef88c11247529376decbf76640f4" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 45603 } } peers { permanent_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38039 } } peers { permanent_uuid: "2e543c72d124494ea1776aea0142f575" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38653 } }
I20260929 19:02:59.550405 10454 leader_election.cc:288] T e0f451a7220d414084f8c5515de8c9d5 P bfeeef88c11247529376decbf76640f4 [CANDIDATE]: Term 1 election: Requested vote from peers 40549454a29b43d1a62d6d9fbdc5a7cf (127.8.158.66:38039), 2e543c72d124494ea1776aea0142f575 (127.8.158.67:38653)
I20260929 19:02:59.550545 10237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f451a7220d414084f8c5515de8c9d5" candidate_uuid: "bfeeef88c11247529376decbf76640f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf"
I20260929 19:02:59.550611 10237 raft_consensus.cc:3037] T e0f451a7220d414084f8c5515de8c9d5 P 40549454a29b43d1a62d6d9fbdc5a7cf [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:59.550603 10377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f451a7220d414084f8c5515de8c9d5" candidate_uuid: "bfeeef88c11247529376decbf76640f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e543c72d124494ea1776aea0142f575"
I20260929 19:02:59.550696 10377 raft_consensus.cc:3037] T e0f451a7220d414084f8c5515de8c9d5 P 2e543c72d124494ea1776aea0142f575 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:59.551290 10377 raft_consensus.cc:2454] T e0f451a7220d414084f8c5515de8c9d5 P 2e543c72d124494ea1776aea0142f575 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfeeef88c11247529376decbf76640f4 in term 1.
I20260929 19:02:59.551290 10237 raft_consensus.cc:2454] T e0f451a7220d414084f8c5515de8c9d5 P 40549454a29b43d1a62d6d9fbdc5a7cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfeeef88c11247529376decbf76640f4 in term 1.
I20260929 19:02:59.551465 10031 leader_election.cc:302] T e0f451a7220d414084f8c5515de8c9d5 P bfeeef88c11247529376decbf76640f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40549454a29b43d1a62d6d9fbdc5a7cf, bfeeef88c11247529376decbf76640f4; no voters:
I20260929 19:02:59.551578 10454 raft_consensus.cc:2781] T e0f451a7220d414084f8c5515de8c9d5 P bfeeef88c11247529376decbf76640f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:02:59.551684 10454 raft_consensus.cc:686] T e0f451a7220d414084f8c5515de8c9d5 P bfeeef88c11247529376decbf76640f4 [term 1 LEADER]: Becoming Leader. State: Replica: bfeeef88c11247529376decbf76640f4, State: Running, Role: LEADER
I20260929 19:02:59.551745 10454 consensus_queue.cc:237] T e0f451a7220d414084f8c5515de8c9d5 P bfeeef88c11247529376decbf76640f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfeeef88c11247529376decbf76640f4" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 45603 } } peers { permanent_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38039 } } peers { permanent_uuid: "2e543c72d124494ea1776aea0142f575" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38653 } }
I20260929 19:02:59.552374 9940 catalog_manager.cc:5495] T e0f451a7220d414084f8c5515de8c9d5 P bfeeef88c11247529376decbf76640f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to bfeeef88c11247529376decbf76640f4 (127.8.158.65). New cstate: current_term: 1 leader_uuid: "bfeeef88c11247529376decbf76640f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfeeef88c11247529376decbf76640f4" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 45603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e543c72d124494ea1776aea0142f575" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38653 } health_report { overall_health: UNKNOWN } } }
W20260929 19:02:59.564000 10144 tablet.cc:2369] T e0f451a7220d414084f8c5515de8c9d5 P bfeeef88c11247529376decbf76640f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 19:02:59 dist-test-slave-nmsz krb5kdc[9895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708578, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:02:59.586874 10482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:59.579027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:45077 (local address 127.0.0.1:59464)
0929 19:02:59.579193 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:59.579206 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:59.579281 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:59.579428 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:59.579429 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:59.579814 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:59.579866 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:59.579868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.580541 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.580542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:59.581375 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:59.581377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.581726 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.581727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:59.581782 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:59.582381 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:02:59.582387 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:59.583977 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:59.585654 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:59.585660 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:59.585662 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:59.585920 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:59.586119 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:59.586120 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:59.586120 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:59.586190 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:59.586474 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:59.586480 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:59.586529 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 19:02:59.586609 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 19:02:59.586750 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":90,"threads_started":1}
I20260929 19:02:59.586882 9989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:59.579106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59464 (local address 127.8.158.126:45077)
0929 19:02:59.579154 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:59.579156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:59.579327 (+ 171us) server_negotiation.cc:407] Connection header received
0929 19:02:59.579351 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:59.579352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:59.579362 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:59.579404 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:59.579967 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.580462 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.581465 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.581638 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.581693 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:59.584090 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:59.584091 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:59.584091 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:59.584098 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:59.585565 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:59.585994 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:59.585994 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:59.585995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:59.586052 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:59.586257 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:59.586258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:59.586258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:59.586411 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:59.586464 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:59.586659 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 19:02:59.586810 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":27}
I20260929 19:02:59.589171 9940 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:59464:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 19:02:59.589421 9940 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:02:59.590806 10357 tablet_service.cc:1467] Processing CreateTablet for tablet 1f2330a489004ff5b4f721fcb691a6ab (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5220d08b6f9349ed9ebc49f0e63c13c8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 19:02:59.590880 10217 tablet_service.cc:1467] Processing CreateTablet for tablet 1f2330a489004ff5b4f721fcb691a6ab (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5220d08b6f9349ed9ebc49f0e63c13c8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 19:02:59.590951 10357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f2330a489004ff5b4f721fcb691a6ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:59.591006 10217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f2330a489004ff5b4f721fcb691a6ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:59.592401 10446 tablet_bootstrap.cc:492] T 1f2330a489004ff5b4f721fcb691a6ab P 2e543c72d124494ea1776aea0142f575: Bootstrap starting.
I20260929 19:02:59.592478 10077 tablet_service.cc:1467] Processing CreateTablet for tablet 1f2330a489004ff5b4f721fcb691a6ab (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5220d08b6f9349ed9ebc49f0e63c13c8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 19:02:59.592608 10077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f2330a489004ff5b4f721fcb691a6ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:59.592842 10446 tablet_bootstrap.cc:654] T 1f2330a489004ff5b4f721fcb691a6ab P 2e543c72d124494ea1776aea0142f575: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:59.593168 10447 tablet_bootstrap.cc:492] T 1f2330a489004ff5b4f721fcb691a6ab P 40549454a29b43d1a62d6d9fbdc5a7cf: Bootstrap starting.
I20260929 19:02:59.593603 10447 tablet_bootstrap.cc:654] T 1f2330a489004ff5b4f721fcb691a6ab P 40549454a29b43d1a62d6d9fbdc5a7cf: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:59.594045 10449 tablet_bootstrap.cc:492] T 1f2330a489004ff5b4f721fcb691a6ab P bfeeef88c11247529376decbf76640f4: Bootstrap starting.
I20260929 19:02:59.594076 10446 tablet_bootstrap.cc:492] T 1f2330a489004ff5b4f721fcb691a6ab P 2e543c72d124494ea1776aea0142f575: No bootstrap required, opened a new log
I20260929 19:02:59.594115 10446 ts_tablet_manager.cc:1397] T 1f2330a489004ff5b4f721fcb691a6ab P 2e543c72d124494ea1776aea0142f575: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:02:59.594257 10446 raft_consensus.cc:348] T 1f2330a489004ff5b4f721fcb691a6ab P 2e543c72d124494ea1776aea0142f575 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfeeef88c11247529376decbf76640f4" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 45603 } } peers { permanent_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38039 } } peers { permanent_uuid: "2e543c72d124494ea1776aea0142f575" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38653 } }
I20260929 19:02:59.594362 10446 raft_consensus.cc:374] T 1f2330a489004ff5b4f721fcb691a6ab P 2e543c72d124494ea1776aea0142f575 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:59.594391 10446 raft_consensus.cc:729] T 1f2330a489004ff5b4f721fcb691a6ab P 2e543c72d124494ea1776aea0142f575 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e543c72d124494ea1776aea0142f575, State: Initialized, Role: FOLLOWER
I20260929 19:02:59.594429 10446 consensus_queue.cc:260] T 1f2330a489004ff5b4f721fcb691a6ab P 2e543c72d124494ea1776aea0142f575 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfeeef88c11247529376decbf76640f4" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 45603 } } peers { permanent_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38039 } } peers { permanent_uuid: "2e543c72d124494ea1776aea0142f575" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38653 } }
I20260929 19:02:59.594447 10449 tablet_bootstrap.cc:654] T 1f2330a489004ff5b4f721fcb691a6ab P bfeeef88c11247529376decbf76640f4: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:59.594512 10446 ts_tablet_manager.cc:1428] T 1f2330a489004ff5b4f721fcb691a6ab P 2e543c72d124494ea1776aea0142f575: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 19:02:59.594609 10452 tablet_replica.cc:439] T 1f2330a489004ff5b4f721fcb691a6ab P 2e543c72d124494ea1776aea0142f575: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfeeef88c11247529376decbf76640f4" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 45603 } } peers { permanent_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38039 } } peers { permanent_uuid: "2e543c72d124494ea1776aea0142f575" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38653 } } }
I20260929 19:02:59.594676 10452 tablet_replica.cc:442] T 1f2330a489004ff5b4f721fcb691a6ab P 2e543c72d124494ea1776aea0142f575: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 19:02:59.595077 10447 tablet_bootstrap.cc:492] T 1f2330a489004ff5b4f721fcb691a6ab P 40549454a29b43d1a62d6d9fbdc5a7cf: No bootstrap required, opened a new log
I20260929 19:02:59.595124 10447 ts_tablet_manager.cc:1397] T 1f2330a489004ff5b4f721fcb691a6ab P 40549454a29b43d1a62d6d9fbdc5a7cf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:02:59.595224 10449 tablet_bootstrap.cc:492] T 1f2330a489004ff5b4f721fcb691a6ab P bfeeef88c11247529376decbf76640f4: No bootstrap required, opened a new log
I20260929 19:02:59.595288 10449 ts_tablet_manager.cc:1397] T 1f2330a489004ff5b4f721fcb691a6ab P bfeeef88c11247529376decbf76640f4: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 19:02:59.595273 10447 raft_consensus.cc:348] T 1f2330a489004ff5b4f721fcb691a6ab P 40549454a29b43d1a62d6d9fbdc5a7cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfeeef88c11247529376decbf76640f4" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 45603 } } peers { permanent_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38039 } } peers { permanent_uuid: "2e543c72d124494ea1776aea0142f575" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38653 } }
I20260929 19:02:59.595335 10447 raft_consensus.cc:374] T 1f2330a489004ff5b4f721fcb691a6ab P 40549454a29b43d1a62d6d9fbdc5a7cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:59.595368 10447 raft_consensus.cc:729] T 1f2330a489004ff5b4f721fcb691a6ab P 40549454a29b43d1a62d6d9fbdc5a7cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40549454a29b43d1a62d6d9fbdc5a7cf, State: Initialized, Role: FOLLOWER
I20260929 19:02:59.595408 10447 consensus_queue.cc:260] T 1f2330a489004ff5b4f721fcb691a6ab P 40549454a29b43d1a62d6d9fbdc5a7cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfeeef88c11247529376decbf76640f4" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 45603 } } peers { permanent_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38039 } } peers { permanent_uuid: "2e543c72d124494ea1776aea0142f575" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38653 } }
I20260929 19:02:59.595443 10449 raft_consensus.cc:348] T 1f2330a489004ff5b4f721fcb691a6ab P bfeeef88c11247529376decbf76640f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfeeef88c11247529376decbf76640f4" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 45603 } } peers { permanent_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38039 } } peers { permanent_uuid: "2e543c72d124494ea1776aea0142f575" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38653 } }
I20260929 19:02:59.595501 10447 ts_tablet_manager.cc:1428] T 1f2330a489004ff5b4f721fcb691a6ab P 40549454a29b43d1a62d6d9fbdc5a7cf: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 19:02:59.595525 10449 raft_consensus.cc:374] T 1f2330a489004ff5b4f721fcb691a6ab P bfeeef88c11247529376decbf76640f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:59.595553 10449 raft_consensus.cc:729] T 1f2330a489004ff5b4f721fcb691a6ab P bfeeef88c11247529376decbf76640f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfeeef88c11247529376decbf76640f4, State: Initialized, Role: FOLLOWER
I20260929 19:02:59.595600 10449 consensus_queue.cc:260] T 1f2330a489004ff5b4f721fcb691a6ab P bfeeef88c11247529376decbf76640f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfeeef88c11247529376decbf76640f4" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 45603 } } peers { permanent_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38039 } } peers { permanent_uuid: "2e543c72d124494ea1776aea0142f575" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38653 } }
I20260929 19:02:59.595613 10453 tablet_replica.cc:439] T 1f2330a489004ff5b4f721fcb691a6ab P 40549454a29b43d1a62d6d9fbdc5a7cf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfeeef88c11247529376decbf76640f4" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 45603 } } peers { permanent_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38039 } } peers { permanent_uuid: "2e543c72d124494ea1776aea0142f575" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38653 } } }
I20260929 19:02:59.595685 10449 ts_tablet_manager.cc:1428] T 1f2330a489004ff5b4f721fcb691a6ab P bfeeef88c11247529376decbf76640f4: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260929 19:02:59.595702 10453 tablet_replica.cc:442] T 1f2330a489004ff5b4f721fcb691a6ab P 40549454a29b43d1a62d6d9fbdc5a7cf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 19:02:59.595898 10475 tablet_replica.cc:439] T 1f2330a489004ff5b4f721fcb691a6ab P bfeeef88c11247529376decbf76640f4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfeeef88c11247529376decbf76640f4" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 45603 } } peers { permanent_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38039 } } peers { permanent_uuid: "2e543c72d124494ea1776aea0142f575" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38653 } } }
I20260929 19:02:59.595988 10475 tablet_replica.cc:442] T 1f2330a489004ff5b4f721fcb691a6ab P bfeeef88c11247529376decbf76640f4: This TxnStatusTablet replica's current role is: FOLLOWER
W20260929 19:02:59.644042 10424 tablet.cc:2369] T 1f2330a489004ff5b4f721fcb691a6ab P 2e543c72d124494ea1776aea0142f575: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:02:59.685974 10475 raft_consensus.cc:482] T 1f2330a489004ff5b4f721fcb691a6ab P bfeeef88c11247529376decbf76640f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:02:59.686100 10475 raft_consensus.cc:504] T 1f2330a489004ff5b4f721fcb691a6ab P bfeeef88c11247529376decbf76640f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfeeef88c11247529376decbf76640f4" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 45603 } } peers { permanent_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38039 } } peers { permanent_uuid: "2e543c72d124494ea1776aea0142f575" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38653 } }
I20260929 19:02:59.686308 10475 leader_election.cc:288] T 1f2330a489004ff5b4f721fcb691a6ab P bfeeef88c11247529376decbf76640f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40549454a29b43d1a62d6d9fbdc5a7cf (127.8.158.66:38039), 2e543c72d124494ea1776aea0142f575 (127.8.158.67:38653)
I20260929 19:02:59.686529 10237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f2330a489004ff5b4f721fcb691a6ab" candidate_uuid: "bfeeef88c11247529376decbf76640f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf" is_pre_election: true
I20260929 19:02:59.686523 10377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f2330a489004ff5b4f721fcb691a6ab" candidate_uuid: "bfeeef88c11247529376decbf76640f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e543c72d124494ea1776aea0142f575" is_pre_election: true
I20260929 19:02:59.686638 10237 raft_consensus.cc:2454] T 1f2330a489004ff5b4f721fcb691a6ab P 40549454a29b43d1a62d6d9fbdc5a7cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfeeef88c11247529376decbf76640f4 in term 0.
I20260929 19:02:59.686636 10377 raft_consensus.cc:2454] T 1f2330a489004ff5b4f721fcb691a6ab P 2e543c72d124494ea1776aea0142f575 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfeeef88c11247529376decbf76640f4 in term 0.
I20260929 19:02:59.686787 10031 leader_election.cc:302] T 1f2330a489004ff5b4f721fcb691a6ab P bfeeef88c11247529376decbf76640f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40549454a29b43d1a62d6d9fbdc5a7cf, bfeeef88c11247529376decbf76640f4; no voters:
I20260929 19:02:59.686934 10475 raft_consensus.cc:2781] T 1f2330a489004ff5b4f721fcb691a6ab P bfeeef88c11247529376decbf76640f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:02:59.686996 10475 raft_consensus.cc:482] T 1f2330a489004ff5b4f721fcb691a6ab P bfeeef88c11247529376decbf76640f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:02:59.687028 10475 raft_consensus.cc:3037] T 1f2330a489004ff5b4f721fcb691a6ab P bfeeef88c11247529376decbf76640f4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:59.687747 10475 raft_consensus.cc:504] T 1f2330a489004ff5b4f721fcb691a6ab P bfeeef88c11247529376decbf76640f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfeeef88c11247529376decbf76640f4" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 45603 } } peers { permanent_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38039 } } peers { permanent_uuid: "2e543c72d124494ea1776aea0142f575" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38653 } }
I20260929 19:02:59.687897 10475 leader_election.cc:288] T 1f2330a489004ff5b4f721fcb691a6ab P bfeeef88c11247529376decbf76640f4 [CANDIDATE]: Term 1 election: Requested vote from peers 40549454a29b43d1a62d6d9fbdc5a7cf (127.8.158.66:38039), 2e543c72d124494ea1776aea0142f575 (127.8.158.67:38653)
I20260929 19:02:59.688046 10377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f2330a489004ff5b4f721fcb691a6ab" candidate_uuid: "bfeeef88c11247529376decbf76640f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e543c72d124494ea1776aea0142f575"
I20260929 19:02:59.688108 10377 raft_consensus.cc:3037] T 1f2330a489004ff5b4f721fcb691a6ab P 2e543c72d124494ea1776aea0142f575 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:59.688076 10237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f2330a489004ff5b4f721fcb691a6ab" candidate_uuid: "bfeeef88c11247529376decbf76640f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf"
I20260929 19:02:59.688158 10237 raft_consensus.cc:3037] T 1f2330a489004ff5b4f721fcb691a6ab P 40549454a29b43d1a62d6d9fbdc5a7cf [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:59.688680 10237 raft_consensus.cc:2454] T 1f2330a489004ff5b4f721fcb691a6ab P 40549454a29b43d1a62d6d9fbdc5a7cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfeeef88c11247529376decbf76640f4 in term 1.
I20260929 19:02:59.688702 10377 raft_consensus.cc:2454] T 1f2330a489004ff5b4f721fcb691a6ab P 2e543c72d124494ea1776aea0142f575 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfeeef88c11247529376decbf76640f4 in term 1.
I20260929 19:02:59.688838 10031 leader_election.cc:302] T 1f2330a489004ff5b4f721fcb691a6ab P bfeeef88c11247529376decbf76640f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40549454a29b43d1a62d6d9fbdc5a7cf, bfeeef88c11247529376decbf76640f4; no voters:
I20260929 19:02:59.688933 10475 raft_consensus.cc:2781] T 1f2330a489004ff5b4f721fcb691a6ab P bfeeef88c11247529376decbf76640f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:02:59.688997 10475 raft_consensus.cc:686] T 1f2330a489004ff5b4f721fcb691a6ab P bfeeef88c11247529376decbf76640f4 [term 1 LEADER]: Becoming Leader. State: Replica: bfeeef88c11247529376decbf76640f4, State: Running, Role: LEADER
I20260929 19:02:59.689031 10475 consensus_queue.cc:237] T 1f2330a489004ff5b4f721fcb691a6ab P bfeeef88c11247529376decbf76640f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfeeef88c11247529376decbf76640f4" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 45603 } } peers { permanent_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38039 } } peers { permanent_uuid: "2e543c72d124494ea1776aea0142f575" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38653 } }
I20260929 19:02:59.689332 10454 tablet_replica.cc:439] T 1f2330a489004ff5b4f721fcb691a6ab P bfeeef88c11247529376decbf76640f4: TxnStatusTablet state changed. Reason: New leader bfeeef88c11247529376decbf76640f4. Latest consensus state: current_term: 1 leader_uuid: "bfeeef88c11247529376decbf76640f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfeeef88c11247529376decbf76640f4" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 45603 } } peers { permanent_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38039 } } peers { permanent_uuid: "2e543c72d124494ea1776aea0142f575" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38653 } } }
I20260929 19:02:59.689458 10454 tablet_replica.cc:442] T 1f2330a489004ff5b4f721fcb691a6ab P bfeeef88c11247529376decbf76640f4: This TxnStatusTablet replica's current role is: LEADER
I20260929 19:02:59.689677 10487 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 19:02:59.689640 9940 catalog_manager.cc:5495] T 1f2330a489004ff5b4f721fcb691a6ab P bfeeef88c11247529376decbf76640f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to bfeeef88c11247529376decbf76640f4 (127.8.158.65). New cstate: current_term: 1 leader_uuid: "bfeeef88c11247529376decbf76640f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfeeef88c11247529376decbf76640f4" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 45603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e543c72d124494ea1776aea0142f575" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38653 } health_report { overall_health: UNKNOWN } } }
I20260929 19:02:59.689796 10487 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 19:02:59.713904 10445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:59.711443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48132 (local address 127.8.158.65:45603)
0929 19:02:59.711493 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:59.711496 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:59.711504 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:02:59.711569 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:59.711570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:59.711586 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:59.711669 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:02:59.711810 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.712296 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.713098 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.713292 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.713377 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:59.713560 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:02:59.713686 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:02:59.713764 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:59.713817 (+ 53us) server_negotiation.cc:299] Negotiation successful
0929 19:02:59.713836 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260929 19:02:59.713930 10482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:59.711357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:45603 (local address 127.0.0.1:48132)
0929 19:02:59.711428 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:59.711441 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:59.711492 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:59.711674 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:59.711675 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:59.711676 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 19:02:59.711727 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:59.711730 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.712380 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.712381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:59.712979 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:59.712981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.713378 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.713379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:59.713444 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:59.713465 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 19:02:59.713755 (+ 290us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 19:02:59.713764 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 19:02:59.713813 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 19:02:59.713859 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 19:02:59.715041 10237 raft_consensus.cc:1261] T 1f2330a489004ff5b4f721fcb691a6ab P 40549454a29b43d1a62d6d9fbdc5a7cf [term 1 FOLLOWER]: Refusing update from remote peer bfeeef88c11247529376decbf76640f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:02:59.715061 10377 raft_consensus.cc:1261] T 1f2330a489004ff5b4f721fcb691a6ab P 2e543c72d124494ea1776aea0142f575 [term 1 FOLLOWER]: Refusing update from remote peer bfeeef88c11247529376decbf76640f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:02:59.715298 10454 consensus_queue.cc:1035] T 1f2330a489004ff5b4f721fcb691a6ab P bfeeef88c11247529376decbf76640f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:02:59.715451 10475 consensus_queue.cc:1035] T 1f2330a489004ff5b4f721fcb691a6ab P bfeeef88c11247529376decbf76640f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e543c72d124494ea1776aea0142f575" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:02:59.715859 10453 tablet_replica.cc:439] T 1f2330a489004ff5b4f721fcb691a6ab P 40549454a29b43d1a62d6d9fbdc5a7cf: TxnStatusTablet state changed. Reason: New leader bfeeef88c11247529376decbf76640f4. Latest consensus state: current_term: 1 leader_uuid: "bfeeef88c11247529376decbf76640f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfeeef88c11247529376decbf76640f4" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 45603 } } peers { permanent_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38039 } } peers { permanent_uuid: "2e543c72d124494ea1776aea0142f575" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38653 } } }
I20260929 19:02:59.715950 10453 tablet_replica.cc:442] T 1f2330a489004ff5b4f721fcb691a6ab P 40549454a29b43d1a62d6d9fbdc5a7cf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 19:02:59.716013 10452 tablet_replica.cc:439] T 1f2330a489004ff5b4f721fcb691a6ab P 2e543c72d124494ea1776aea0142f575: TxnStatusTablet state changed. Reason: New leader bfeeef88c11247529376decbf76640f4. Latest consensus state: current_term: 1 leader_uuid: "bfeeef88c11247529376decbf76640f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfeeef88c11247529376decbf76640f4" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 45603 } } peers { permanent_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38039 } } peers { permanent_uuid: "2e543c72d124494ea1776aea0142f575" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38653 } } }
I20260929 19:02:59.716099 10452 tablet_replica.cc:442] T 1f2330a489004ff5b4f721fcb691a6ab P 2e543c72d124494ea1776aea0142f575: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 19:02:59.716457 10475 tablet_replica.cc:439] T 1f2330a489004ff5b4f721fcb691a6ab P bfeeef88c11247529376decbf76640f4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bfeeef88c11247529376decbf76640f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfeeef88c11247529376decbf76640f4" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 45603 } } peers { permanent_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38039 } } peers { permanent_uuid: "2e543c72d124494ea1776aea0142f575" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38653 } } }
I20260929 19:02:59.716538 10475 tablet_replica.cc:442] T 1f2330a489004ff5b4f721fcb691a6ab P bfeeef88c11247529376decbf76640f4: This TxnStatusTablet replica's current role is: LEADER
I20260929 19:02:59.716595 10461 mvcc.cc:205] Tried to move back new op lower bound from 7334742342511005696 to 7334742342406770688. Current Snapshot: MvccSnapshot[applied={T|T < 7334742342511005696}]
I20260929 19:02:59.717041 10475 tablet_replica.cc:439] T 1f2330a489004ff5b4f721fcb691a6ab P bfeeef88c11247529376decbf76640f4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bfeeef88c11247529376decbf76640f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfeeef88c11247529376decbf76640f4" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 45603 } } peers { permanent_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38039 } } peers { permanent_uuid: "2e543c72d124494ea1776aea0142f575" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38653 } } }
I20260929 19:02:59.717123 10475 tablet_replica.cc:442] T 1f2330a489004ff5b4f721fcb691a6ab P bfeeef88c11247529376decbf76640f4: This TxnStatusTablet replica's current role is: LEADER
I20260929 19:02:59.717166 10460 mvcc.cc:205] Tried to move back new op lower bound from 7334742342511005696 to 7334742342406770688. Current Snapshot: MvccSnapshot[applied={T|T < 7334742342511005696}]
I20260929 19:02:59.717278 10453 tablet_replica.cc:439] T 1f2330a489004ff5b4f721fcb691a6ab P 40549454a29b43d1a62d6d9fbdc5a7cf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bfeeef88c11247529376decbf76640f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfeeef88c11247529376decbf76640f4" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 45603 } } peers { permanent_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38039 } } peers { permanent_uuid: "2e543c72d124494ea1776aea0142f575" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38653 } } }
I20260929 19:02:59.717334 10453 tablet_replica.cc:442] T 1f2330a489004ff5b4f721fcb691a6ab P 40549454a29b43d1a62d6d9fbdc5a7cf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 19:02:59.718219 10452 tablet_replica.cc:439] T 1f2330a489004ff5b4f721fcb691a6ab P 2e543c72d124494ea1776aea0142f575: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bfeeef88c11247529376decbf76640f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfeeef88c11247529376decbf76640f4" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 45603 } } peers { permanent_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38039 } } peers { permanent_uuid: "2e543c72d124494ea1776aea0142f575" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38653 } } }
I20260929 19:02:59.718333 10452 tablet_replica.cc:442] T 1f2330a489004ff5b4f721fcb691a6ab P 2e543c72d124494ea1776aea0142f575: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 19:02:59.720894 10445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:59.718651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48136 (local address 127.8.158.65:45603)
0929 19:02:59.718689 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:59.718691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:59.718699 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:02:59.718718 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:59.718719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:59.718728 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:59.718789 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:02:59.718902 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.719318 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.720337 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.720499 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.720550 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:59.720695 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:02:59.720717 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:02:59.720763 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:59.720817 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 19:02:59.720831 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 19:02:59.725504 10445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:59.723092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:46845 (local address 127.8.158.65:45603)
0929 19:02:59.723130 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:59.723132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:59.723285 (+ 153us) server_negotiation.cc:407] Connection header received
0929 19:02:59.723306 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:59.723307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:59.723314 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:59.723357 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:02:59.723468 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.723865 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.724965 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.725122 (+ 157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.725163 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:59.725343 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:02:59.725362 (+ 19us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:02:59.725400 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:59.725451 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 19:02:59.725465 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 19:02:59.725606 10494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:59.723023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:45603 (local address 127.8.158.65:46845)
0929 19:02:59.723196 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:59.723206 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:59.723264 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:59.723362 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:59.723362 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:59.723364 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 19:02:59.723414 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:59.723415 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.723931 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.723932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:59.724884 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:59.724886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.725182 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.725183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:59.725254 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:59.725267 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 19:02:59.725401 (+ 134us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 19:02:59.725402 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 19:02:59.725450 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 19:02:59.725491 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":106,"threads_started":1}
I20260929 19:02:59.727385 10377 raft_consensus.cc:1261] T e0f451a7220d414084f8c5515de8c9d5 P 2e543c72d124494ea1776aea0142f575 [term 1 FOLLOWER]: Refusing update from remote peer bfeeef88c11247529376decbf76640f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:02:59.727438 10237 raft_consensus.cc:1261] T e0f451a7220d414084f8c5515de8c9d5 P 40549454a29b43d1a62d6d9fbdc5a7cf [term 1 FOLLOWER]: Refusing update from remote peer bfeeef88c11247529376decbf76640f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:02:59.727581 10495 consensus_queue.cc:1035] T e0f451a7220d414084f8c5515de8c9d5 P bfeeef88c11247529376decbf76640f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e543c72d124494ea1776aea0142f575" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:02:59.727653 10454 consensus_queue.cc:1035] T e0f451a7220d414084f8c5515de8c9d5 P bfeeef88c11247529376decbf76640f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40549454a29b43d1a62d6d9fbdc5a7cf" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:02:59.744763 9940 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:02:59.744854 9940 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:02:59.745476 9940 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 869266291b584c3583320a409b4e2c9c: Sending DeleteTablet for 3 replicas of tablet e0f451a7220d414084f8c5515de8c9d5
I20260929 19:02:59.745715 10217 tablet_service.cc:1514] Processing DeleteTablet for tablet e0f451a7220d414084f8c5515de8c9d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:02:59 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:42282
I20260929 19:02:59.745710 10357 tablet_service.cc:1514] Processing DeleteTablet for tablet e0f451a7220d414084f8c5515de8c9d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:02:59 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:43994
I20260929 19:02:59.745800 10469 tablet_replica.cc:330] T e0f451a7220d414084f8c5515de8c9d5 P 40549454a29b43d1a62d6d9fbdc5a7cf: stopping tablet replica
I20260929 19:02:59.745800 10468 tablet_replica.cc:330] T e0f451a7220d414084f8c5515de8c9d5 P 2e543c72d124494ea1776aea0142f575: stopping tablet replica
I20260929 19:02:59.745841 10469 raft_consensus.cc:2229] T e0f451a7220d414084f8c5515de8c9d5 P 40549454a29b43d1a62d6d9fbdc5a7cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:02:59.745846 10468 raft_consensus.cc:2229] T e0f451a7220d414084f8c5515de8c9d5 P 2e543c72d124494ea1776aea0142f575 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:02:59.745872 10469 raft_consensus.cc:2258] T e0f451a7220d414084f8c5515de8c9d5 P 40549454a29b43d1a62d6d9fbdc5a7cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:02:59.745877 10468 raft_consensus.cc:2258] T e0f451a7220d414084f8c5515de8c9d5 P 2e543c72d124494ea1776aea0142f575 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:02:59.746032 10077 tablet_service.cc:1514] Processing DeleteTablet for tablet e0f451a7220d414084f8c5515de8c9d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:02:59 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:48128
I20260929 19:02:59.746131 10470 tablet_replica.cc:330] T e0f451a7220d414084f8c5515de8c9d5 P bfeeef88c11247529376decbf76640f4: stopping tablet replica
I20260929 19:02:59.746187 10470 raft_consensus.cc:2229] T e0f451a7220d414084f8c5515de8c9d5 P bfeeef88c11247529376decbf76640f4 [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:02:59.746264 10470 raft_consensus.cc:2258] T e0f451a7220d414084f8c5515de8c9d5 P bfeeef88c11247529376decbf76640f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:02:59.746644 10470 ts_tablet_manager.cc:1905] T e0f451a7220d414084f8c5515de8c9d5 P bfeeef88c11247529376decbf76640f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:02:59.746723 10469 ts_tablet_manager.cc:1905] T e0f451a7220d414084f8c5515de8c9d5 P 40549454a29b43d1a62d6d9fbdc5a7cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:02:59.747828 10468 ts_tablet_manager.cc:1905] T e0f451a7220d414084f8c5515de8c9d5 P 2e543c72d124494ea1776aea0142f575: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:02:59.748008 10469 ts_tablet_manager.cc:1918] T e0f451a7220d414084f8c5515de8c9d5 P 40549454a29b43d1a62d6d9fbdc5a7cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 19:02:59.748054 10469 log.cc:1196] T e0f451a7220d414084f8c5515de8c9d5 P 40549454a29b43d1a62d6d9fbdc5a7cf: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-1/wal/wals/e0f451a7220d414084f8c5515de8c9d5
I20260929 19:02:59.748173 10470 ts_tablet_manager.cc:1918] T e0f451a7220d414084f8c5515de8c9d5 P bfeeef88c11247529376decbf76640f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 19:02:59.748220 10470 log.cc:1196] T e0f451a7220d414084f8c5515de8c9d5 P bfeeef88c11247529376decbf76640f4: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-0/wal/wals/e0f451a7220d414084f8c5515de8c9d5
I20260929 19:02:59.748255 10469 ts_tablet_manager.cc:1939] T e0f451a7220d414084f8c5515de8c9d5 P 40549454a29b43d1a62d6d9fbdc5a7cf: Deleting consensus metadata
I20260929 19:02:59.748440 10470 ts_tablet_manager.cc:1939] T e0f451a7220d414084f8c5515de8c9d5 P bfeeef88c11247529376decbf76640f4: Deleting consensus metadata
I20260929 19:02:59.748523 9927 catalog_manager.cc:4841] TS 40549454a29b43d1a62d6d9fbdc5a7cf (127.8.158.66:38039): tablet e0f451a7220d414084f8c5515de8c9d5 (table test-table [id=6ce7c1fd4f77460f80805196fd09954f]) successfully deleted
I20260929 19:02:59.748775 9927 catalog_manager.cc:4841] TS bfeeef88c11247529376decbf76640f4 (127.8.158.65:45603): tablet e0f451a7220d414084f8c5515de8c9d5 (table test-table [id=6ce7c1fd4f77460f80805196fd09954f]) successfully deleted
I20260929 19:02:59.749222 10468 ts_tablet_manager.cc:1918] T e0f451a7220d414084f8c5515de8c9d5 P 2e543c72d124494ea1776aea0142f575: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 19:02:59.749279 10468 log.cc:1196] T e0f451a7220d414084f8c5515de8c9d5 P 2e543c72d124494ea1776aea0142f575: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708573736807-8825-0/minicluster-data/ts-2/wal/wals/e0f451a7220d414084f8c5515de8c9d5
I20260929 19:02:59.749472 10468 ts_tablet_manager.cc:1939] T e0f451a7220d414084f8c5515de8c9d5 P 2e543c72d124494ea1776aea0142f575: Deleting consensus metadata
I20260929 19:02:59.749799 9925 catalog_manager.cc:4841] TS 2e543c72d124494ea1776aea0142f575 (127.8.158.67:38653): tablet e0f451a7220d414084f8c5515de8c9d5 (table test-table [id=6ce7c1fd4f77460f80805196fd09954f]) successfully deleted
Sep 29 19:02:59 dist-test-slave-nmsz krb5kdc[9895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708579, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.158.65@KRBTEST.COM
I20260929 19:02:59.758862 10445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:59.746453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48142 (local address 127.8.158.65:45603)
0929 19:02:59.746488 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:59.746490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:59.750491 (+ 4001us) server_negotiation.cc:407] Connection header received
0929 19:02:59.750517 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:59.750518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:59.750527 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:59.750596 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:59.751085 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.751811 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.752517 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.752805 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.752877 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:59.755945 (+ 3068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:59.755952 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:59.755953 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:59.755961 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:59.757763 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:59.758088 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:59.758091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:59.758091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:59.758151 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:59.758357 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:59.758358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:59.758358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:59.758504 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:59.758644 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:59.758702 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 19:02:59.758776 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260929 19:02:59.759048 10137 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48142
I20260929 19:02:59.766261 9989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:59.759953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59480 (local address 127.8.158.126:45077)
0929 19:02:59.760000 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:59.760002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:59.760028 (+ 26us) server_negotiation.cc:407] Connection header received
0929 19:02:59.760087 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:59.760088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:59.760099 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:59.760151 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:59.760576 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.761328 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.762001 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:59.762280 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:59.762407 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:59.763462 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:59.763464 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:59.763464 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:59.763471 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:59.765001 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:59.765332 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:59.765334 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:59.765334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:59.765404 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:59.765713 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:59.765713 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:59.765714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:59.765844 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:59.765940 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:59.766092 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 19:02:59.766168 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260929 19:02:59.766502 9940 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59480
I20260929 19:02:59.767055 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 10012
I20260929 19:02:59.772697 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 10152
I20260929 19:02:59.777295 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 10292
I20260929 19:02:59.782035 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 9908
2026-09-29T19:02:59Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2240 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:02:59 dist-test-slave-nmsz krb5kdc[10505](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:02:59 dist-test-slave-nmsz krb5kdc[10505](info): set up 2 sockets
Sep 29 19:02:59 dist-test-slave-nmsz krb5kdc[10505](info): commencing operation
krb5kdc: starting...
W20260929 19:03:00.814019 8825 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Sep 29 19:03:00 dist-test-slave-nmsz krb5kdc[10505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:03:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:03:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.158.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
I20260929 19:03:00.929265 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:33825
--webserver_interface=127.8.158.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:42045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:33825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:01.008581 10518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:01.008780 10518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:01.008801 10518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:01.009858 10518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:03:01.009891 10518 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 19:03:01.009904 10518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:01.009917 10518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:01.009930 10518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:03:01.009941 10518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:03:01.011322 10518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:42045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:33825
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:33825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.158.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:01.011507 10518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:01.011664 10518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:01.016533 10526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:01.016664 10518 server_base.cc:1034] running on GCE node
W20260929 19:03:01.016592 10523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:01.016580 10524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:01.017040 10518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:01.017338 10518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:01.018509 10518 hybrid_clock.cc:648] HybridClock initialized: now 1790708581018464 us; error 61 us; skew 500 ppm
Sep 29 19:03:01 dist-test-slave-nmsz krb5kdc[10505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708581, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:01.020912 10518 init.cc:377] Logged in from keytab as kudu/127.8.158.126@KRBTEST.COM (short username kudu)
I20260929 19:03:01.021270 10518 webserver.cc:460] Webserver started at http://127.8.158.126:39679/ using document root <none> and password file <none>
I20260929 19:03:01.021435 10518 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:01.021472 10518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:01.021555 10518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:01.022601 10518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "06a804e1e1b34f37b46d8763e45660ad"
format_stamp: "Formatted at 2026-09-29 19:03:01 on dist-test-slave-nmsz"
server_key: "1b285f5517e8d8d783b027de952761ae"
server_key_iv: "0f5df9de0972a499f5e1003f368f0a26"
server_key_version: "encryptionkey@0"
I20260929 19:03:01.022900 10518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "06a804e1e1b34f37b46d8763e45660ad"
format_stamp: "Formatted at 2026-09-29 19:03:01 on dist-test-slave-nmsz"
server_key: "1b285f5517e8d8d783b027de952761ae"
server_key_iv: "0f5df9de0972a499f5e1003f368f0a26"
server_key_version: "encryptionkey@0"
I20260929 19:03:01.024159 10518 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:03:01.024809 10533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:01.025002 10518 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:03:01.025089 10518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "06a804e1e1b34f37b46d8763e45660ad"
format_stamp: "Formatted at 2026-09-29 19:03:01 on dist-test-slave-nmsz"
server_key: "1b285f5517e8d8d783b027de952761ae"
server_key_iv: "0f5df9de0972a499f5e1003f368f0a26"
server_key_version: "encryptionkey@0"
I20260929 19:03:01.025139 10518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:01.056564 10518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:01.058853 10518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:01.058965 10518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:01.063360 10518 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.126:33825
I20260929 19:03:01.063447 10595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.126:33825 every 8 connection(s)
I20260929 19:03:01.063755 10518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:03:01.063901 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 10518
I20260929 19:03:01.063993 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:03:01.064184 8825 external_mini_cluster.cc:1428] Setting key 3102757f3dc2f2fda99a0df4bf0d4b84
I20260929 19:03:01.064466 10596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:01.067685 10596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06a804e1e1b34f37b46d8763e45660ad: Bootstrap starting.
I20260929 19:03:01.068274 10596 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06a804e1e1b34f37b46d8763e45660ad: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:01.068588 10596 log.cc:824] T 00000000000000000000000000000000 P 06a804e1e1b34f37b46d8763e45660ad: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:01.069294 10596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06a804e1e1b34f37b46d8763e45660ad: No bootstrap required, opened a new log
Sep 29 19:03:01 dist-test-slave-nmsz krb5kdc[10505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:03:01.071277 10596 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06a804e1e1b34f37b46d8763e45660ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a804e1e1b34f37b46d8763e45660ad" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 33825 } }
I20260929 19:03:01.071455 10596 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06a804e1e1b34f37b46d8763e45660ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:01.071504 10596 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06a804e1e1b34f37b46d8763e45660ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06a804e1e1b34f37b46d8763e45660ad, State: Initialized, Role: FOLLOWER
I20260929 19:03:01.071640 10596 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06a804e1e1b34f37b46d8763e45660ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a804e1e1b34f37b46d8763e45660ad" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 33825 } }
I20260929 19:03:01.071731 10596 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06a804e1e1b34f37b46d8763e45660ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:03:01.071768 10596 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06a804e1e1b34f37b46d8763e45660ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:03:01.071820 10596 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06a804e1e1b34f37b46d8763e45660ad [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:01.072525 10596 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06a804e1e1b34f37b46d8763e45660ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a804e1e1b34f37b46d8763e45660ad" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 33825 } }
I20260929 19:03:01.072641 10596 leader_election.cc:302] T 00000000000000000000000000000000 P 06a804e1e1b34f37b46d8763e45660ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06a804e1e1b34f37b46d8763e45660ad; no voters:
I20260929 19:03:01.072791 10596 leader_election.cc:288] T 00000000000000000000000000000000 P 06a804e1e1b34f37b46d8763e45660ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:03:01.072860 10601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06a804e1e1b34f37b46d8763e45660ad [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:03:01.073022 10596 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06a804e1e1b34f37b46d8763e45660ad [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:03:01.073195 10601 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06a804e1e1b34f37b46d8763e45660ad [term 1 LEADER]: Becoming Leader. State: Replica: 06a804e1e1b34f37b46d8763e45660ad, State: Running, Role: LEADER
I20260929 19:03:01.073318 10601 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06a804e1e1b34f37b46d8763e45660ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a804e1e1b34f37b46d8763e45660ad" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 33825 } }
I20260929 19:03:01.073660 10601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06a804e1e1b34f37b46d8763e45660ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06a804e1e1b34f37b46d8763e45660ad. Latest consensus state: current_term: 1 leader_uuid: "06a804e1e1b34f37b46d8763e45660ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a804e1e1b34f37b46d8763e45660ad" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 33825 } } }
I20260929 19:03:01.073817 10601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06a804e1e1b34f37b46d8763e45660ad [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:01.073930 10603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06a804e1e1b34f37b46d8763e45660ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06a804e1e1b34f37b46d8763e45660ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a804e1e1b34f37b46d8763e45660ad" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 33825 } } }
I20260929 19:03:01.074018 10603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06a804e1e1b34f37b46d8763e45660ad [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:01.074412 10610 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:03:01.075090 10610 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:03:01.075333 10599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:01.064715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35870 (local address 127.8.158.126:33825)
0929 19:03:01.064901 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:01.064910 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:01.064946 (+ 36us) server_negotiation.cc:407] Connection header received
0929 19:03:01.065572 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:01.065581 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:01.065639 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:01.065790 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:01.066348 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.067682 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.068479 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.069028 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.069137 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:01.071393 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:01.071397 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:01.071397 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:01.071408 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:01.073409 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:01.073895 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:01.073898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:01.073898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:01.074007 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:01.074257 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:01.074258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:01.074258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:01.074604 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:01.074753 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:01.075005 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 19:03:01.075140 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":88,"threads_started":1}
I20260929 19:03:01.076858 10610 catalog_manager.cc:1347] Generated new cluster ID: ad373a8bb1954c2abbf79ded1462f6d9
I20260929 19:03:01.076907 10610 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:03:01.102237 10610 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:03:01.102912 10610 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:03:01.110239 10610 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06a804e1e1b34f37b46d8763e45660ad: Generated new TSK 0
I20260929 19:03:01.110436 10610 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.158.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab.
Entry for principal kudu/127.8.158.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab.
I20260929 19:03:01.136778 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.65:0
--local_ip_for_outbound_sockets=127.8.158.65
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:33825
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab
--principal=kudu/127.8.158.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:42045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:01.216229 10622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:01.216445 10622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:01.216487 10622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:01.217815 10622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:01.217883 10622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:01.217939 10622 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 19:03:01.217963 10622 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 19:03:01.218001 10622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.65
I20260929 19:03:01.219465 10622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:42045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab
--principal=kudu/127.8.158.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.158.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.158.126:33825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.65
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:01.219722 10622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:01.219940 10622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:01.224951 10627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:01.224956 10630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:01.225003 10628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:01.225270 10622 server_base.cc:1034] running on GCE node
I20260929 19:03:01.225466 10622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:01.225728 10622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:01.226895 10622 hybrid_clock.cc:648] HybridClock initialized: now 1790708581226885 us; error 31 us; skew 500 ppm
Sep 29 19:03:01 dist-test-slave-nmsz krb5kdc[10505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708581, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:01.229506 10622 init.cc:377] Logged in from keytab as kudu/127.8.158.65@KRBTEST.COM (short username kudu)
I20260929 19:03:01.229930 10622 webserver.cc:460] Webserver started at http://127.8.158.65:39787/ using document root <none> and password file <none>
I20260929 19:03:01.230158 10622 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:01.230226 10622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:01.230403 10622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:01.231508 10622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-0/data/instance:
uuid: "5dce99a6cd474c6abad05507d315d1e5"
format_stamp: "Formatted at 2026-09-29 19:03:01 on dist-test-slave-nmsz"
server_key: "27ba039cea2e89a3aca3eef749a24f34"
server_key_iv: "17981d222b181b8de0964a3f2c1af52e"
server_key_version: "encryptionkey@0"
I20260929 19:03:01.231861 10622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance:
uuid: "5dce99a6cd474c6abad05507d315d1e5"
format_stamp: "Formatted at 2026-09-29 19:03:01 on dist-test-slave-nmsz"
server_key: "27ba039cea2e89a3aca3eef749a24f34"
server_key_iv: "17981d222b181b8de0964a3f2c1af52e"
server_key_version: "encryptionkey@0"
I20260929 19:03:01.233225 10622 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:03:01.234207 10637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:01.234452 10622 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:01.234557 10622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-0/wal
uuid: "5dce99a6cd474c6abad05507d315d1e5"
format_stamp: "Formatted at 2026-09-29 19:03:01 on dist-test-slave-nmsz"
server_key: "27ba039cea2e89a3aca3eef749a24f34"
server_key_iv: "17981d222b181b8de0964a3f2c1af52e"
server_key_version: "encryptionkey@0"
I20260929 19:03:01.234643 10622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:01.266211 10622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:01.268567 10622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:01.268729 10622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:01.269256 10622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:01.269318 10622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:01.269380 10622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:01.269412 10622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:01.276284 10622 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.65:33741
I20260929 19:03:01.276347 10752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.65:33741 every 8 connection(s)
Sep 29 19:03:01 dist-test-slave-nmsz krb5kdc[10505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708581, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:03:01.276656 10622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
I20260929 19:03:01.279551 10599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:01.269788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:56277 (local address 127.8.158.126:33825)
0929 19:03:01.269836 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:01.269838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:01.270051 (+ 213us) server_negotiation.cc:407] Connection header received
0929 19:03:01.270639 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:01.270641 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:01.270654 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:01.270705 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:01.271560 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.272394 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.274004 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.274392 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.274477 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:01.276895 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:01.276897 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:01.276897 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:01.276904 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:01.278388 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:01.278774 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:01.278775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:01.278775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:01.278833 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:01.279087 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:01.279088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:01.279088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:01.279191 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:01.279243 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:01.279418 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 19:03:01.279483 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 19:03:01.279767 10646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:01.269787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:33825 (local address 127.8.158.65:56277)
0929 19:03:01.269928 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:01.269961 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:01.270525 (+ 564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:01.270775 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:01.270778 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:01.271156 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:01.271448 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:01.271456 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.272508 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.272509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:01.273928 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:01.273931 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.274478 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.274479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:01.274582 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:01.275163 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:01.275172 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:01.276784 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:01.278457 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:01.278462 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:01.278464 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:01.278679 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:01.278903 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:01.278904 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:01.278904 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:01.279028 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:01.279233 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:01.279235 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:01.279334 (+ 99us) client_negotiation.cc:769] Sending connection context
0929 19:03:01.279406 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 19:03:01.279521 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":84,"threads_started":1}
I20260929 19:03:01.280293 10753 heartbeater.cc:347] Connected to a master server at 127.8.158.126:33825
I20260929 19:03:01.280375 10753 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:01.280570 10753 heartbeater.cc:511] Master 127.8.158.126:33825 requested a full tablet report, sending...
I20260929 19:03:01.281121 10549 ts_manager.cc:194] Registered new tserver with Master: 5dce99a6cd474c6abad05507d315d1e5 (127.8.158.65:33741)
I20260929 19:03:01.281786 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 10622
I20260929 19:03:01.281863 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance
I20260929 19:03:01.281988 10549 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.65@KRBTEST.COM'} at 127.8.158.65:56277
I20260929 19:03:01.282030 8825 external_mini_cluster.cc:1428] Setting key 0d9029b6c004a3898689c4dd6388651e
I20260929 19:03:01.287544 10759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:01.281326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:33825 (local address 127.8.158.65:33855)
0929 19:03:01.281471 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:01.281487 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:01.281529 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:01.282306 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:01.282307 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:01.282521 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:01.282577 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:01.282579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.283302 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.283302 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:01.283772 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:01.283773 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.284159 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.284160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:01.284241 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:01.284658 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:01.284661 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:01.284908 (+ 247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:01.286512 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:01.286513 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:01.286513 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:01.286696 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:01.286911 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:01.286912 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:01.286912 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:01.286969 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:01.287221 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:01.287222 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:01.287266 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 19:03:01.287352 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 19:03:01.287450 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":85,"threads_started":1}
I20260929 19:03:01.287544 10599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:01.281326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:33855 (local address 127.8.158.126:33825)
0929 19:03:01.282192 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:01.282194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:01.282202 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:03:01.282218 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:01.282219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:01.282228 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:01.282283 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:01.282673 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.283209 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.283855 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.284067 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.284156 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:01.285066 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:01.285068 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:01.285068 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:01.285075 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:01.286406 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:01.286783 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:01.286784 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:01.286784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:01.286839 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:01.287044 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:01.287045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:01.287045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:01.287154 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:01.287229 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:01.287376 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 19:03:01.287449 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":849}
WARNING: no policy specified for kudu/127.8.158.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab.
Entry for principal kudu/127.8.158.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab.
I20260929 19:03:01.306363 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.66:0
--local_ip_for_outbound_sockets=127.8.158.66
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:33825
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab
--principal=kudu/127.8.158.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:42045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:01.386433 10762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:01.386615 10762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:01.386637 10762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:01.387683 10762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:01.387715 10762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:01.387735 10762 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 19:03:01.387748 10762 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 19:03:01.387764 10762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.66
I20260929 19:03:01.389447 10762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:42045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab
--principal=kudu/127.8.158.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.158.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.158.126:33825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.66
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:01.389628 10762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:01.389804 10762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 19:03:01.395088 10762 server_base.cc:1034] running on GCE node
W20260929 19:03:01.395123 10768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:01.395257 10767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:01.395102 10770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:01.395571 10762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:01.395838 10762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:01.397044 10762 hybrid_clock.cc:648] HybridClock initialized: now 1790708581397024 us; error 39 us; skew 500 ppm
Sep 29 19:03:01 dist-test-slave-nmsz krb5kdc[10505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708581, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:01.399611 10762 init.cc:377] Logged in from keytab as kudu/127.8.158.66@KRBTEST.COM (short username kudu)
I20260929 19:03:01.400043 10762 webserver.cc:460] Webserver started at http://127.8.158.66:44257/ using document root <none> and password file <none>
I20260929 19:03:01.400273 10762 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:01.400316 10762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:01.400400 10762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:01.401335 10762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-1/data/instance:
uuid: "1094d0c4c0524fc7b6308499947dddb2"
format_stamp: "Formatted at 2026-09-29 19:03:01 on dist-test-slave-nmsz"
server_key: "1dde6d475690a6fc8b728d394ed1f0ab"
server_key_iv: "ff0f68d53f61046699b341fdc3d107ec"
server_key_version: "encryptionkey@0"
I20260929 19:03:01.401628 10762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance:
uuid: "1094d0c4c0524fc7b6308499947dddb2"
format_stamp: "Formatted at 2026-09-29 19:03:01 on dist-test-slave-nmsz"
server_key: "1dde6d475690a6fc8b728d394ed1f0ab"
server_key_iv: "ff0f68d53f61046699b341fdc3d107ec"
server_key_version: "encryptionkey@0"
I20260929 19:03:01.402911 10762 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260929 19:03:01.403612 10777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:01.403805 10762 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:03:01.403883 10762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-1/wal
uuid: "1094d0c4c0524fc7b6308499947dddb2"
format_stamp: "Formatted at 2026-09-29 19:03:01 on dist-test-slave-nmsz"
server_key: "1dde6d475690a6fc8b728d394ed1f0ab"
server_key_iv: "ff0f68d53f61046699b341fdc3d107ec"
server_key_version: "encryptionkey@0"
I20260929 19:03:01.403934 10762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:01.417451 10762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:01.419296 10762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:01.419399 10762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:01.419819 10762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:01.419855 10762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:01.419907 10762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:01.419934 10762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:01.426378 10762 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.66:46157
I20260929 19:03:01.426445 10892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.66:46157 every 8 connection(s)
I20260929 19:03:01.426714 10762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
Sep 29 19:03:01 dist-test-slave-nmsz krb5kdc[10505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708581, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.66@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:03:01.429862 10599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:01.420290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:33513 (local address 127.8.158.126:33825)
0929 19:03:01.420380 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:01.420382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:01.420629 (+ 247us) server_negotiation.cc:407] Connection header received
0929 19:03:01.421174 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:01.421175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:01.421191 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:01.421264 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:01.422098 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.422958 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.424489 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.424751 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.424837 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:01.427303 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:01.427304 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:01.427305 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:01.427310 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:01.428657 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:01.429038 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:01.429039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:01.429039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:01.429087 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:01.429297 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:01.429297 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:01.429298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:01.429408 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:01.429460 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:01.429741 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 19:03:01.429800 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 19:03:01.430068 10786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:01.420257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:33825 (local address 127.8.158.66:33513)
0929 19:03:01.420527 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:01.420554 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:01.421039 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:01.421343 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:01.421347 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:01.421673 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:01.421993 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:01.421999 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.423070 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.423071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:01.424399 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:01.424403 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.424844 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.424845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:01.424993 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:01.425490 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:01.425498 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:01.427215 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:01.428756 (+ 1541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:01.428761 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:01.428763 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:01.428963 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:01.429149 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:01.429149 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:01.429150 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:01.429241 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:01.429503 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:01.429508 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:01.429616 (+ 108us) client_negotiation.cc:769] Sending connection context
0929 19:03:01.429711 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 19:03:01.429823 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":122,"threads_started":1}
I20260929 19:03:01.430711 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 10762
I20260929 19:03:01.430723 10893 heartbeater.cc:347] Connected to a master server at 127.8.158.126:33825
I20260929 19:03:01.430805 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance
I20260929 19:03:01.430811 10893 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:01.430998 10893 heartbeater.cc:511] Master 127.8.158.126:33825 requested a full tablet report, sending...
I20260929 19:03:01.431008 8825 external_mini_cluster.cc:1428] Setting key 37f4476d7cba8cd6a158a71364fbda81
I20260929 19:03:01.431618 10550 ts_manager.cc:194] Registered new tserver with Master: 1094d0c4c0524fc7b6308499947dddb2 (127.8.158.66:46157)
I20260929 19:03:01.432142 10550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.66@KRBTEST.COM'} at 127.8.158.66:33513
I20260929 19:03:01.439368 10599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:01.433167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:57863 (local address 127.8.158.126:33825)
0929 19:03:01.433232 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:01.433234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:01.433243 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:03:01.433265 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:01.433265 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:01.433276 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:01.433334 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:01.433940 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.434670 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.435539 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.435733 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.435818 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:01.436744 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:01.436745 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:01.436746 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:01.436752 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:01.438204 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:01.438581 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:01.438582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:01.438583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:01.438636 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:01.438834 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:01.438835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:01.438835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:01.438983 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:01.439064 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:01.439209 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 19:03:01.439281 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 19:03:01.439375 10899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:01.431805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:33825 (local address 127.8.158.66:57863)
0929 19:03:01.431954 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:01.431964 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:01.432029 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:01.433395 (+ 1366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:01.433396 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:01.433695 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:01.433798 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:01.433800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.434919 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.434920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:01.435428 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:01.435429 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.435819 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.435820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:01.435890 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:01.436344 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:01.436348 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:01.436666 (+ 318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:01.438331 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:01.438332 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:01.438332 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:01.438498 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:01.438708 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:01.438709 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:01.438710 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:01.438772 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:01.439055 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:01.439056 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:01.439099 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 19:03:01.439186 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 19:03:01.439283 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":107,"threads_started":1}
WARNING: no policy specified for kudu/127.8.158.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab.
Entry for principal kudu/127.8.158.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab.
I20260929 19:03:01.456965 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.67:0
--local_ip_for_outbound_sockets=127.8.158.67
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:33825
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab
--principal=kudu/127.8.158.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:42045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:01.535760 10902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:01.535945 10902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:01.535969 10902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:01.537034 10902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:01.537070 10902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:01.537091 10902 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 19:03:01.537102 10902 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 19:03:01.537118 10902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.67
I20260929 19:03:01.538792 10902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:42045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab
--principal=kudu/127.8.158.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.158.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.158.126:33825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.67
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:01.538966 10902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:01.539129 10902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:01.544130 10910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:01.544114 10908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:01.544210 10902 server_base.cc:1034] running on GCE node
W20260929 19:03:01.544114 10907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:01.544502 10902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:01.544751 10902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:01.545940 10902 hybrid_clock.cc:648] HybridClock initialized: now 1790708581545875 us; error 106 us; skew 500 ppm
Sep 29 19:03:01 dist-test-slave-nmsz krb5kdc[10505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708581, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:01.548421 10902 init.cc:377] Logged in from keytab as kudu/127.8.158.67@KRBTEST.COM (short username kudu)
I20260929 19:03:01.548839 10902 webserver.cc:460] Webserver started at http://127.8.158.67:41775/ using document root <none> and password file <none>
I20260929 19:03:01.549067 10902 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:01.549135 10902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:01.549279 10902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:01.550254 10902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-2/data/instance:
uuid: "1b41ba80cad24d3db1d428f488d903d2"
format_stamp: "Formatted at 2026-09-29 19:03:01 on dist-test-slave-nmsz"
server_key: "af05fd10e5e2e76564d547468c387617"
server_key_iv: "e93a622be0dde4297c7a7e409c32431a"
server_key_version: "encryptionkey@0"
I20260929 19:03:01.550668 10902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance:
uuid: "1b41ba80cad24d3db1d428f488d903d2"
format_stamp: "Formatted at 2026-09-29 19:03:01 on dist-test-slave-nmsz"
server_key: "af05fd10e5e2e76564d547468c387617"
server_key_iv: "e93a622be0dde4297c7a7e409c32431a"
server_key_version: "encryptionkey@0"
I20260929 19:03:01.551924 10902 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:01.552634 10917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:01.552974 10902 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:01.553107 10902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-2/wal
uuid: "1b41ba80cad24d3db1d428f488d903d2"
format_stamp: "Formatted at 2026-09-29 19:03:01 on dist-test-slave-nmsz"
server_key: "af05fd10e5e2e76564d547468c387617"
server_key_iv: "e93a622be0dde4297c7a7e409c32431a"
server_key_version: "encryptionkey@0"
I20260929 19:03:01.553201 10902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:01.567801 10902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:01.569638 10902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:01.569789 10902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:01.570245 10902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:01.570308 10902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:01.570410 10902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:01.570441 10902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:01.577560 10902 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.67:38517
Sep 29 19:03:01 dist-test-slave-nmsz krb5kdc[10505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708581, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.67@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:03:01.577663 11032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.67:38517 every 8 connection(s)
I20260929 19:03:01.577920 10902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
I20260929 19:03:01.580845 10599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:01.570739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:51323 (local address 127.8.158.126:33825)
0929 19:03:01.570805 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:01.570807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:01.571082 (+ 275us) server_negotiation.cc:407] Connection header received
0929 19:03:01.571505 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:01.571514 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:01.571527 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:01.571609 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:01.572542 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.573419 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.575008 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.575349 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.575427 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:01.578090 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:01.578092 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:01.578092 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:01.578100 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:01.579592 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:01.579966 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:01.579967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:01.579967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:01.580025 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:01.580296 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:01.580297 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:01.580297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:01.580397 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:01.580460 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:01.580719 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 19:03:01.580785 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 19:03:01.581106 10926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:01.570734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:33825 (local address 127.8.158.67:51323)
0929 19:03:01.570956 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:01.570996 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:01.571419 (+ 423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:01.571722 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:01.571727 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:01.572148 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:01.572424 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:01.572429 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.573517 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.573518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:01.574904 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:01.574907 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.575523 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.575524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:01.575679 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:01.576185 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:01.576196 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:01.577985 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:01.579674 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:01.579678 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:01.579680 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:01.579881 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:01.580096 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:01.580097 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:01.580097 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:01.580234 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:01.580439 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:01.580458 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:01.580596 (+ 138us) client_negotiation.cc:769] Sending connection context
0929 19:03:01.580698 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 19:03:01.580831 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":106,"threads_started":1}
I20260929 19:03:01.581137 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 10902
I20260929 19:03:01.581228 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance
I20260929 19:03:01.581408 8825 external_mini_cluster.cc:1428] Setting key 852fd73acfc8cd4f4eff6d6ca6125c3d
I20260929 19:03:01.582379 11033 heartbeater.cc:347] Connected to a master server at 127.8.158.126:33825
I20260929 19:03:01.582463 11033 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:01.582633 11033 heartbeater.cc:511] Master 127.8.158.126:33825 requested a full tablet report, sending...
I20260929 19:03:01.583006 10549 ts_manager.cc:194] Registered new tserver with Master: 1b41ba80cad24d3db1d428f488d903d2 (127.8.158.67:38517)
I20260929 19:03:01.583454 10549 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.67@KRBTEST.COM'} at 127.8.158.67:51323
I20260929 19:03:01.587723 10599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:01.582670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:43327 (local address 127.8.158.126:33825)
0929 19:03:01.582708 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:01.582709 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:01.582829 (+ 120us) server_negotiation.cc:407] Connection header received
0929 19:03:01.582873 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:01.582874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:01.582883 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:01.582944 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:01.583256 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.583828 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.584639 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.584808 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.584861 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:01.585646 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:01.585647 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:01.585647 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:01.585653 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:01.586861 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:01.587203 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:01.587204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:01.587204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:01.587251 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:01.587399 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:01.587399 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:01.587399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:01.587477 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:01.587520 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:01.587615 (+ 95us) server_negotiation.cc:299] Negotiation successful
0929 19:03:01.587668 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 19:03:01.587728 11039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:01.582615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:33825 (local address 127.8.158.67:43327)
0929 19:03:01.582756 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:01.582766 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:01.582824 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:01.582950 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:01.582951 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:01.583136 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:01.583190 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:01.583192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.583921 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.583922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:01.584561 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:01.584563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.584873 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.584874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:01.584923 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:01.585311 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:01.585315 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:01.585572 (+ 257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:01.586938 (+ 1366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:01.586938 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:01.586939 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:01.587135 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:01.587310 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:01.587311 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:01.587311 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:01.587361 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:01.587523 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:01.587524 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:01.587556 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 19:03:01.587602 (+ 46us) client_negotiation.cc:240] Negotiation successful
0929 19:03:01.587671 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":74,"threads_started":1}
I20260929 19:03:01.592873 8825 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 19:03:01 dist-test-slave-nmsz krb5kdc[10505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 19:03:01 dist-test-slave-nmsz krb5kdc[10505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:03:01.615249 10599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:01.607708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35874 (local address 127.8.158.126:33825)
0929 19:03:01.607765 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:01.607766 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:01.607775 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:03:01.607831 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:01.607832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:01.607840 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:01.607887 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:01.608415 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.608881 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.609608 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.609791 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.609848 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:01.612472 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:01.612474 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:01.612475 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:01.612482 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:01.614178 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:01.614605 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:01.614606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:01.614607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:01.614662 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:01.614846 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:01.614846 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:01.614847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:01.614944 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:01.615025 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:01.615132 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 19:03:01.615187 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 19:03:01.616940 10549 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:03:01.618531 10549 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:03:01.625452 11050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:01.621451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.67:38517 (local address 127.0.0.1:36660)
0929 19:03:01.621554 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:01.621594 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:01.621679 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:01.622079 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:01.622082 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:01.622085 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:01.622151 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:01.622153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.623262 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.623263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:01.624280 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:01.624283 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.625163 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.625164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:01.625239 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:01.625253 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 19:03:01.625330 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 19:03:01.625378 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":65,"threads_started":1}
I20260929 19:03:01.625473 11053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:01.621701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36660 (local address 127.8.158.67:38517)
0929 19:03:01.621887 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:01.621891 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:01.621902 (+ 11us) server_negotiation.cc:407] Connection header received
0929 19:03:01.621941 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:01.621943 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:01.621958 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:01.622229 (+ 271us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:01.622283 (+ 54us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.623179 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.624352 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.625074 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.625155 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:01.625215 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:01.625352 (+ 137us) server_negotiation.cc:299] Negotiation successful
0929 19:03:01.625388 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":84,"threads_started":1}
I20260929 19:03:01.626144 11051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:01.621556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:33741 (local address 127.0.0.1:49858)
0929 19:03:01.621919 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:01.621931 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:01.621982 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:01.622530 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:01.622531 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:01.622533 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:01.622586 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:01.622587 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.623868 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.623869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:01.624758 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:01.624760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.625898 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.625899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:01.625978 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:01.625982 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:03:01.626034 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 19:03:01.626071 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":44,"thread_start_us":85,"threads_started":1}
I20260929 19:03:01.626170 10967 tablet_service.cc:1467] Processing CreateTablet for tablet bde57a8452e6447a964bc097b88da77c (DEFAULT_TABLE table=test-table [id=5f3d6556a184479d9ba59213cebc17d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:01.626436 11054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:01.621771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45686 (local address 127.8.158.66:46157)
0929 19:03:01.621929 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:01.621933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:01.622273 (+ 340us) server_negotiation.cc:407] Connection header received
0929 19:03:01.622340 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:01.622343 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:01.622358 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:01.622541 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:01.622780 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.623829 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.625185 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.625948 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.626030 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:01.626082 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:01.626195 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 19:03:01.626336 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":69,"threads_started":1}
I20260929 19:03:01.626610 10967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bde57a8452e6447a964bc097b88da77c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:01.626670 11052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:01.621556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.66:46157 (local address 127.0.0.1:45686)
0929 19:03:01.622159 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:01.622169 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:01.622217 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:01.622671 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:01.622671 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:01.622673 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:01.622712 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:01.622713 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.623912 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.623913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:01.625097 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:01.625100 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.626032 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.626033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:01.626113 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:01.626117 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:03:01.626182 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 19:03:01.626454 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":3,"thread_start_us":78,"threads_started":1}
I20260929 19:03:01.626746 11055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:01.621945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49858 (local address 127.8.158.65:33741)
0929 19:03:01.622208 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:01.622213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:01.622223 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:03:01.622261 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:01.622262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:01.622293 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:01.622525 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:01.622661 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.623777 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.624842 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.625470 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.625873 (+ 403us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:01.625935 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:01.626257 (+ 322us) server_negotiation.cc:299] Negotiation successful
0929 19:03:01.626635 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":111,"threads_started":1}
I20260929 19:03:01.627300 10827 tablet_service.cc:1467] Processing CreateTablet for tablet bde57a8452e6447a964bc097b88da77c (DEFAULT_TABLE table=test-table [id=5f3d6556a184479d9ba59213cebc17d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:01.627426 10687 tablet_service.cc:1467] Processing CreateTablet for tablet bde57a8452e6447a964bc097b88da77c (DEFAULT_TABLE table=test-table [id=5f3d6556a184479d9ba59213cebc17d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:01.627647 10827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bde57a8452e6447a964bc097b88da77c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:01.627749 10687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bde57a8452e6447a964bc097b88da77c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:01.629897 11056 tablet_bootstrap.cc:492] T bde57a8452e6447a964bc097b88da77c P 1b41ba80cad24d3db1d428f488d903d2: Bootstrap starting.
I20260929 19:03:01.630654 11057 tablet_bootstrap.cc:492] T bde57a8452e6447a964bc097b88da77c P 1094d0c4c0524fc7b6308499947dddb2: Bootstrap starting.
I20260929 19:03:01.631011 11056 tablet_bootstrap.cc:654] T bde57a8452e6447a964bc097b88da77c P 1b41ba80cad24d3db1d428f488d903d2: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:01.631270 11057 tablet_bootstrap.cc:654] T bde57a8452e6447a964bc097b88da77c P 1094d0c4c0524fc7b6308499947dddb2: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:01.631347 11056 log.cc:824] T bde57a8452e6447a964bc097b88da77c P 1b41ba80cad24d3db1d428f488d903d2: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:01.631616 11058 tablet_bootstrap.cc:492] T bde57a8452e6447a964bc097b88da77c P 5dce99a6cd474c6abad05507d315d1e5: Bootstrap starting.
I20260929 19:03:01.631665 11057 log.cc:824] T bde57a8452e6447a964bc097b88da77c P 1094d0c4c0524fc7b6308499947dddb2: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:01.632258 11058 tablet_bootstrap.cc:654] T bde57a8452e6447a964bc097b88da77c P 5dce99a6cd474c6abad05507d315d1e5: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:01.632393 11057 tablet_bootstrap.cc:492] T bde57a8452e6447a964bc097b88da77c P 1094d0c4c0524fc7b6308499947dddb2: No bootstrap required, opened a new log
I20260929 19:03:01.632467 11057 ts_tablet_manager.cc:1397] T bde57a8452e6447a964bc097b88da77c P 1094d0c4c0524fc7b6308499947dddb2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:03:01.632586 11058 log.cc:824] T bde57a8452e6447a964bc097b88da77c P 5dce99a6cd474c6abad05507d315d1e5: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:01.633319 11058 tablet_bootstrap.cc:492] T bde57a8452e6447a964bc097b88da77c P 5dce99a6cd474c6abad05507d315d1e5: No bootstrap required, opened a new log
I20260929 19:03:01.633319 11056 tablet_bootstrap.cc:492] T bde57a8452e6447a964bc097b88da77c P 1b41ba80cad24d3db1d428f488d903d2: No bootstrap required, opened a new log
I20260929 19:03:01.633391 11056 ts_tablet_manager.cc:1397] T bde57a8452e6447a964bc097b88da77c P 1b41ba80cad24d3db1d428f488d903d2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 19:03:01.633391 11058 ts_tablet_manager.cc:1397] T bde57a8452e6447a964bc097b88da77c P 5dce99a6cd474c6abad05507d315d1e5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:03:01.634159 11057 raft_consensus.cc:348] T bde57a8452e6447a964bc097b88da77c P 1094d0c4c0524fc7b6308499947dddb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } } peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } }
I20260929 19:03:01.634346 11057 raft_consensus.cc:374] T bde57a8452e6447a964bc097b88da77c P 1094d0c4c0524fc7b6308499947dddb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:01.634392 11057 raft_consensus.cc:729] T bde57a8452e6447a964bc097b88da77c P 1094d0c4c0524fc7b6308499947dddb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1094d0c4c0524fc7b6308499947dddb2, State: Initialized, Role: FOLLOWER
I20260929 19:03:01.634500 11057 consensus_queue.cc:260] T bde57a8452e6447a964bc097b88da77c P 1094d0c4c0524fc7b6308499947dddb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } } peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } }
I20260929 19:03:01.634791 10893 heartbeater.cc:503] Master 127.8.158.126:33825 was elected leader, sending a full tablet report...
I20260929 19:03:01.634781 11057 ts_tablet_manager.cc:1428] T bde57a8452e6447a964bc097b88da77c P 1094d0c4c0524fc7b6308499947dddb2: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260929 19:03:01.635345 11056 raft_consensus.cc:348] T bde57a8452e6447a964bc097b88da77c P 1b41ba80cad24d3db1d428f488d903d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } } peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } }
I20260929 19:03:01.635345 11058 raft_consensus.cc:348] T bde57a8452e6447a964bc097b88da77c P 5dce99a6cd474c6abad05507d315d1e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } } peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } }
I20260929 19:03:01.635490 11056 raft_consensus.cc:374] T bde57a8452e6447a964bc097b88da77c P 1b41ba80cad24d3db1d428f488d903d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:01.635490 11058 raft_consensus.cc:374] T bde57a8452e6447a964bc097b88da77c P 5dce99a6cd474c6abad05507d315d1e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:01.635541 11056 raft_consensus.cc:729] T bde57a8452e6447a964bc097b88da77c P 1b41ba80cad24d3db1d428f488d903d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b41ba80cad24d3db1d428f488d903d2, State: Initialized, Role: FOLLOWER
I20260929 19:03:01.635542 11058 raft_consensus.cc:729] T bde57a8452e6447a964bc097b88da77c P 5dce99a6cd474c6abad05507d315d1e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dce99a6cd474c6abad05507d315d1e5, State: Initialized, Role: FOLLOWER
I20260929 19:03:01.635655 11058 consensus_queue.cc:260] T bde57a8452e6447a964bc097b88da77c P 5dce99a6cd474c6abad05507d315d1e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } } peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } }
I20260929 19:03:01.635655 11056 consensus_queue.cc:260] T bde57a8452e6447a964bc097b88da77c P 1b41ba80cad24d3db1d428f488d903d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } } peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } }
I20260929 19:03:01.635910 11056 ts_tablet_manager.cc:1428] T bde57a8452e6447a964bc097b88da77c P 1b41ba80cad24d3db1d428f488d903d2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 19:03:01.635910 11058 ts_tablet_manager.cc:1428] T bde57a8452e6447a964bc097b88da77c P 5dce99a6cd474c6abad05507d315d1e5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 19:03:01.636018 11033 heartbeater.cc:503] Master 127.8.158.126:33825 was elected leader, sending a full tablet report...
I20260929 19:03:01.636086 10753 heartbeater.cc:503] Master 127.8.158.126:33825 was elected leader, sending a full tablet report...
W20260929 19:03:01.677268 10894 tablet.cc:2369] T bde57a8452e6447a964bc097b88da77c P 1094d0c4c0524fc7b6308499947dddb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 19:03:01.777302 10754 tablet.cc:2369] T bde57a8452e6447a964bc097b88da77c P 5dce99a6cd474c6abad05507d315d1e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:03:01.809702 11063 raft_consensus.cc:482] T bde57a8452e6447a964bc097b88da77c P 1b41ba80cad24d3db1d428f488d903d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:03:01.809810 11063 raft_consensus.cc:504] T bde57a8452e6447a964bc097b88da77c P 1b41ba80cad24d3db1d428f488d903d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } } peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } }
I20260929 19:03:01.810240 11063 leader_election.cc:288] T bde57a8452e6447a964bc097b88da77c P 1b41ba80cad24d3db1d428f488d903d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dce99a6cd474c6abad05507d315d1e5 (127.8.158.65:33741), 1094d0c4c0524fc7b6308499947dddb2 (127.8.158.66:46157)
I20260929 19:03:01.814124 11055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:01.810529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:33093 (local address 127.8.158.65:33741)
0929 19:03:01.810590 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:01.810591 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:01.810599 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:03:01.810629 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:01.810630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:01.810638 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:01.810706 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:01.810889 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.811651 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.813153 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.813689 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.813775 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:01.813784 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:01.814020 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 19:03:01.814057 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 19:03:01.814129 11054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:01.810528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:36321 (local address 127.8.158.66:46157)
0929 19:03:01.810583 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:01.810585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:01.810718 (+ 133us) server_negotiation.cc:407] Connection header received
0929 19:03:01.810747 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:01.810749 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:01.810758 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:01.810810 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:01.810979 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.811660 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.813153 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.813719 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.813783 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:01.813791 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:01.814020 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 19:03:01.814055 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 19:03:01.814153 10926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:01.810460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:33741 (local address 127.8.158.67:33093)
0929 19:03:01.810516 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:01.810528 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:01.810576 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:01.810715 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:01.810716 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:01.810718 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:01.810809 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:01.810812 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.811736 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.811737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:01.813051 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:01.813054 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.813821 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.813822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:01.813934 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:01.813936 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 19:03:01.814002 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 19:03:01.814050 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":131}
I20260929 19:03:01.814165 11065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:01.810477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.66:46157 (local address 127.8.158.67:36321)
0929 19:03:01.810641 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:01.810650 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:01.810705 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:01.810824 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:01.810825 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:01.810827 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:01.810890 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:01.810892 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.811736 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.811737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:01.813051 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:01.813054 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.813821 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.813822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:01.813922 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:01.813926 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:03:01.814002 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 19:03:01.814050 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":25,"thread_start_us":90,"threads_started":1}
I20260929 19:03:01.814541 10707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bde57a8452e6447a964bc097b88da77c" candidate_uuid: "1b41ba80cad24d3db1d428f488d903d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dce99a6cd474c6abad05507d315d1e5" is_pre_election: true
I20260929 19:03:01.814620 10847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bde57a8452e6447a964bc097b88da77c" candidate_uuid: "1b41ba80cad24d3db1d428f488d903d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1094d0c4c0524fc7b6308499947dddb2" is_pre_election: true
I20260929 19:03:01.814695 10707 raft_consensus.cc:2454] T bde57a8452e6447a964bc097b88da77c P 5dce99a6cd474c6abad05507d315d1e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b41ba80cad24d3db1d428f488d903d2 in term 0.
I20260929 19:03:01.814729 10847 raft_consensus.cc:2454] T bde57a8452e6447a964bc097b88da77c P 1094d0c4c0524fc7b6308499947dddb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b41ba80cad24d3db1d428f488d903d2 in term 0.
I20260929 19:03:01.814924 10919 leader_election.cc:302] T bde57a8452e6447a964bc097b88da77c P 1b41ba80cad24d3db1d428f488d903d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1094d0c4c0524fc7b6308499947dddb2, 1b41ba80cad24d3db1d428f488d903d2, 5dce99a6cd474c6abad05507d315d1e5; no voters:
I20260929 19:03:01.815145 11063 raft_consensus.cc:2781] T bde57a8452e6447a964bc097b88da77c P 1b41ba80cad24d3db1d428f488d903d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:03:01.815219 11063 raft_consensus.cc:482] T bde57a8452e6447a964bc097b88da77c P 1b41ba80cad24d3db1d428f488d903d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:03:01.815256 11063 raft_consensus.cc:3037] T bde57a8452e6447a964bc097b88da77c P 1b41ba80cad24d3db1d428f488d903d2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:01.815958 11063 raft_consensus.cc:504] T bde57a8452e6447a964bc097b88da77c P 1b41ba80cad24d3db1d428f488d903d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } } peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } }
I20260929 19:03:01.816111 11063 leader_election.cc:288] T bde57a8452e6447a964bc097b88da77c P 1b41ba80cad24d3db1d428f488d903d2 [CANDIDATE]: Term 1 election: Requested vote from peers 5dce99a6cd474c6abad05507d315d1e5 (127.8.158.65:33741), 1094d0c4c0524fc7b6308499947dddb2 (127.8.158.66:46157)
I20260929 19:03:01.816246 10707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bde57a8452e6447a964bc097b88da77c" candidate_uuid: "1b41ba80cad24d3db1d428f488d903d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dce99a6cd474c6abad05507d315d1e5"
I20260929 19:03:01.816326 10707 raft_consensus.cc:3037] T bde57a8452e6447a964bc097b88da77c P 5dce99a6cd474c6abad05507d315d1e5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:01.816294 10847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bde57a8452e6447a964bc097b88da77c" candidate_uuid: "1b41ba80cad24d3db1d428f488d903d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1094d0c4c0524fc7b6308499947dddb2"
I20260929 19:03:01.816421 10847 raft_consensus.cc:3037] T bde57a8452e6447a964bc097b88da77c P 1094d0c4c0524fc7b6308499947dddb2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:01.817010 10707 raft_consensus.cc:2454] T bde57a8452e6447a964bc097b88da77c P 5dce99a6cd474c6abad05507d315d1e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b41ba80cad24d3db1d428f488d903d2 in term 1.
I20260929 19:03:01.817119 10847 raft_consensus.cc:2454] T bde57a8452e6447a964bc097b88da77c P 1094d0c4c0524fc7b6308499947dddb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b41ba80cad24d3db1d428f488d903d2 in term 1.
I20260929 19:03:01.817164 10919 leader_election.cc:302] T bde57a8452e6447a964bc097b88da77c P 1b41ba80cad24d3db1d428f488d903d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b41ba80cad24d3db1d428f488d903d2, 5dce99a6cd474c6abad05507d315d1e5; no voters:
I20260929 19:03:01.817250 11063 raft_consensus.cc:2781] T bde57a8452e6447a964bc097b88da77c P 1b41ba80cad24d3db1d428f488d903d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:03:01.817394 11063 raft_consensus.cc:686] T bde57a8452e6447a964bc097b88da77c P 1b41ba80cad24d3db1d428f488d903d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1b41ba80cad24d3db1d428f488d903d2, State: Running, Role: LEADER
I20260929 19:03:01.817533 11063 consensus_queue.cc:237] T bde57a8452e6447a964bc097b88da77c P 1b41ba80cad24d3db1d428f488d903d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } } peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } }
I20260929 19:03:01.818284 10550 catalog_manager.cc:5495] T bde57a8452e6447a964bc097b88da77c P 1b41ba80cad24d3db1d428f488d903d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b41ba80cad24d3db1d428f488d903d2 (127.8.158.67). New cstate: current_term: 1 leader_uuid: "1b41ba80cad24d3db1d428f488d903d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } health_report { overall_health: UNKNOWN } } }
W20260929 19:03:01.828606 11034 tablet.cc:2369] T bde57a8452e6447a964bc097b88da77c P 1b41ba80cad24d3db1d428f488d903d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 19:03:01 dist-test-slave-nmsz krb5kdc[10505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708581, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:03:01.870731 10599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:01.863247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35878 (local address 127.8.158.126:33825)
0929 19:03:01.863318 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:01.863319 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:01.863484 (+ 165us) server_negotiation.cc:407] Connection header received
0929 19:03:01.863513 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:01.863513 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:01.863522 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:01.863589 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:01.864099 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.864566 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.865284 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.865453 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.865508 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:01.868087 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:01.868088 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:01.868088 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:01.868096 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:01.869475 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:01.869871 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:01.869872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:01.869872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:01.869929 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:01.870153 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:01.870153 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:01.870154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:01.870256 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:01.870364 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:01.870520 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 19:03:01.870606 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 19:03:01.870817 11073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:01.863140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:33825 (local address 127.0.0.1:35878)
0929 19:03:01.863398 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:01.863408 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:01.863477 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:01.863611 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:01.863612 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:01.863930 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:01.863988 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:01.863991 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.864653 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.864654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:01.865202 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:01.865204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.865553 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.865554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:01.865611 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:01.866197 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:01.866204 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:01.867982 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:01.869573 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:01.869580 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:01.869582 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:01.869784 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:01.870004 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:01.870005 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:01.870005 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:01.870064 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:01.870353 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:01.870359 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:01.870405 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 19:03:01.870499 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 19:03:01.870680 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":75,"thread_start_us":208,"threads_started":1}
I20260929 19:03:01.873102 10550 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:35878:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 19:03:01.873333 10550 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:03:01.874884 10687 tablet_service.cc:1467] Processing CreateTablet for tablet 414f31ccf3cd48debdb07c61252e4463 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8ac8ddf51923400d9e7f8ef59781c9a7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 19:03:01.874956 10827 tablet_service.cc:1467] Processing CreateTablet for tablet 414f31ccf3cd48debdb07c61252e4463 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8ac8ddf51923400d9e7f8ef59781c9a7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 19:03:01.875054 10687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 414f31ccf3cd48debdb07c61252e4463. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:01.875141 10827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 414f31ccf3cd48debdb07c61252e4463. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:01.876333 11057 tablet_bootstrap.cc:492] T 414f31ccf3cd48debdb07c61252e4463 P 1094d0c4c0524fc7b6308499947dddb2: Bootstrap starting.
I20260929 19:03:01.876413 10967 tablet_service.cc:1467] Processing CreateTablet for tablet 414f31ccf3cd48debdb07c61252e4463 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8ac8ddf51923400d9e7f8ef59781c9a7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 19:03:01.876555 10967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 414f31ccf3cd48debdb07c61252e4463. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:01.876910 11057 tablet_bootstrap.cc:654] T 414f31ccf3cd48debdb07c61252e4463 P 1094d0c4c0524fc7b6308499947dddb2: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:01.877511 11057 tablet_bootstrap.cc:492] T 414f31ccf3cd48debdb07c61252e4463 P 1094d0c4c0524fc7b6308499947dddb2: No bootstrap required, opened a new log
I20260929 19:03:01.877552 11057 ts_tablet_manager.cc:1397] T 414f31ccf3cd48debdb07c61252e4463 P 1094d0c4c0524fc7b6308499947dddb2: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 19:03:01.877677 11057 raft_consensus.cc:348] T 414f31ccf3cd48debdb07c61252e4463 P 1094d0c4c0524fc7b6308499947dddb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } }
I20260929 19:03:01.877732 11057 raft_consensus.cc:374] T 414f31ccf3cd48debdb07c61252e4463 P 1094d0c4c0524fc7b6308499947dddb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:01.877746 11057 raft_consensus.cc:729] T 414f31ccf3cd48debdb07c61252e4463 P 1094d0c4c0524fc7b6308499947dddb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1094d0c4c0524fc7b6308499947dddb2, State: Initialized, Role: FOLLOWER
I20260929 19:03:01.877805 11057 consensus_queue.cc:260] T 414f31ccf3cd48debdb07c61252e4463 P 1094d0c4c0524fc7b6308499947dddb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } }
I20260929 19:03:01.877904 11057 ts_tablet_manager.cc:1428] T 414f31ccf3cd48debdb07c61252e4463 P 1094d0c4c0524fc7b6308499947dddb2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:01.877910 11056 tablet_bootstrap.cc:492] T 414f31ccf3cd48debdb07c61252e4463 P 1b41ba80cad24d3db1d428f488d903d2: Bootstrap starting.
I20260929 19:03:01.877948 11058 tablet_bootstrap.cc:492] T 414f31ccf3cd48debdb07c61252e4463 P 5dce99a6cd474c6abad05507d315d1e5: Bootstrap starting.
I20260929 19:03:01.878396 11058 tablet_bootstrap.cc:654] T 414f31ccf3cd48debdb07c61252e4463 P 5dce99a6cd474c6abad05507d315d1e5: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:01.878413 11056 tablet_bootstrap.cc:654] T 414f31ccf3cd48debdb07c61252e4463 P 1b41ba80cad24d3db1d428f488d903d2: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:01.878809 11062 tablet_replica.cc:439] T 414f31ccf3cd48debdb07c61252e4463 P 1094d0c4c0524fc7b6308499947dddb2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } } }
I20260929 19:03:01.878912 11062 tablet_replica.cc:442] T 414f31ccf3cd48debdb07c61252e4463 P 1094d0c4c0524fc7b6308499947dddb2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 19:03:01.879047 11056 tablet_bootstrap.cc:492] T 414f31ccf3cd48debdb07c61252e4463 P 1b41ba80cad24d3db1d428f488d903d2: No bootstrap required, opened a new log
I20260929 19:03:01.879089 11056 ts_tablet_manager.cc:1397] T 414f31ccf3cd48debdb07c61252e4463 P 1b41ba80cad24d3db1d428f488d903d2: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 19:03:01.879112 11058 tablet_bootstrap.cc:492] T 414f31ccf3cd48debdb07c61252e4463 P 5dce99a6cd474c6abad05507d315d1e5: No bootstrap required, opened a new log
I20260929 19:03:01.879153 11058 ts_tablet_manager.cc:1397] T 414f31ccf3cd48debdb07c61252e4463 P 5dce99a6cd474c6abad05507d315d1e5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 19:03:01.879225 11056 raft_consensus.cc:348] T 414f31ccf3cd48debdb07c61252e4463 P 1b41ba80cad24d3db1d428f488d903d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } }
I20260929 19:03:01.879283 11056 raft_consensus.cc:374] T 414f31ccf3cd48debdb07c61252e4463 P 1b41ba80cad24d3db1d428f488d903d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:01.879307 11056 raft_consensus.cc:729] T 414f31ccf3cd48debdb07c61252e4463 P 1b41ba80cad24d3db1d428f488d903d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b41ba80cad24d3db1d428f488d903d2, State: Initialized, Role: FOLLOWER
I20260929 19:03:01.879292 11058 raft_consensus.cc:348] T 414f31ccf3cd48debdb07c61252e4463 P 5dce99a6cd474c6abad05507d315d1e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } }
I20260929 19:03:01.879359 11058 raft_consensus.cc:374] T 414f31ccf3cd48debdb07c61252e4463 P 5dce99a6cd474c6abad05507d315d1e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:01.879381 11058 raft_consensus.cc:729] T 414f31ccf3cd48debdb07c61252e4463 P 5dce99a6cd474c6abad05507d315d1e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dce99a6cd474c6abad05507d315d1e5, State: Initialized, Role: FOLLOWER
I20260929 19:03:01.879354 11056 consensus_queue.cc:260] T 414f31ccf3cd48debdb07c61252e4463 P 1b41ba80cad24d3db1d428f488d903d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } }
I20260929 19:03:01.879427 11058 consensus_queue.cc:260] T 414f31ccf3cd48debdb07c61252e4463 P 5dce99a6cd474c6abad05507d315d1e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } }
I20260929 19:03:01.879475 11056 ts_tablet_manager.cc:1428] T 414f31ccf3cd48debdb07c61252e4463 P 1b41ba80cad24d3db1d428f488d903d2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:01.879535 11058 ts_tablet_manager.cc:1428] T 414f31ccf3cd48debdb07c61252e4463 P 5dce99a6cd474c6abad05507d315d1e5: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260929 19:03:01.879527 11063 tablet_replica.cc:439] T 414f31ccf3cd48debdb07c61252e4463 P 1b41ba80cad24d3db1d428f488d903d2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } } }
I20260929 19:03:01.879629 11063 tablet_replica.cc:442] T 414f31ccf3cd48debdb07c61252e4463 P 1b41ba80cad24d3db1d428f488d903d2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 19:03:01.879652 11064 tablet_replica.cc:439] T 414f31ccf3cd48debdb07c61252e4463 P 5dce99a6cd474c6abad05507d315d1e5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } } }
I20260929 19:03:01.879710 11064 tablet_replica.cc:442] T 414f31ccf3cd48debdb07c61252e4463 P 5dce99a6cd474c6abad05507d315d1e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 19:03:01.904860 11064 raft_consensus.cc:482] T 414f31ccf3cd48debdb07c61252e4463 P 5dce99a6cd474c6abad05507d315d1e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:03:01.905076 11064 raft_consensus.cc:504] T 414f31ccf3cd48debdb07c61252e4463 P 5dce99a6cd474c6abad05507d315d1e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } }
I20260929 19:03:01.905435 11064 leader_election.cc:288] T 414f31ccf3cd48debdb07c61252e4463 P 5dce99a6cd474c6abad05507d315d1e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1094d0c4c0524fc7b6308499947dddb2 (127.8.158.66:46157), 1b41ba80cad24d3db1d428f488d903d2 (127.8.158.67:38517)
I20260929 19:03:01.909165 11053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:01.905911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:54657 (local address 127.8.158.67:38517)
0929 19:03:01.905973 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:01.905981 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:01.906087 (+ 106us) server_negotiation.cc:407] Connection header received
0929 19:03:01.906219 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:01.906220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:01.906230 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:01.906304 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:01.906554 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.907251 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.908289 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.908830 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.908905 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:01.908914 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:01.909087 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 19:03:01.909114 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 19:03:01.909206 11077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:01.905669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.67:38517 (local address 127.8.158.65:54657)
0929 19:03:01.905950 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:01.905966 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:01.906125 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:01.906370 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:01.906371 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:01.906374 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:01.906474 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:01.906476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.907337 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.907338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:01.908191 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:01.908194 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.908932 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.908933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:01.908982 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:01.908985 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:03:01.909070 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 19:03:01.909108 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":1,"spinlock_wait_cycles":11008,"thread_start_us":125,"threads_started":1}
I20260929 19:03:01.909503 10985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414f31ccf3cd48debdb07c61252e4463" candidate_uuid: "5dce99a6cd474c6abad05507d315d1e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b41ba80cad24d3db1d428f488d903d2" is_pre_election: true
I20260929 19:03:01.909636 10985 raft_consensus.cc:2454] T 414f31ccf3cd48debdb07c61252e4463 P 1b41ba80cad24d3db1d428f488d903d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dce99a6cd474c6abad05507d315d1e5 in term 0.
I20260929 19:03:01.909888 10638 leader_election.cc:302] T 414f31ccf3cd48debdb07c61252e4463 P 5dce99a6cd474c6abad05507d315d1e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b41ba80cad24d3db1d428f488d903d2, 5dce99a6cd474c6abad05507d315d1e5; no voters:
I20260929 19:03:01.909965 11054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:01.905870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:37277 (local address 127.8.158.66:46157)
0929 19:03:01.905950 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:01.905952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:01.905969 (+ 17us) server_negotiation.cc:407] Connection header received
0929 19:03:01.906079 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:01.906081 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:01.906093 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:01.906173 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:01.906401 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.907396 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.909057 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.909596 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.909669 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:01.909677 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:01.909873 (+ 196us) server_negotiation.cc:299] Negotiation successful
0929 19:03:01.909917 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 19:03:01.910005 11078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:01.905667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.66:46157 (local address 127.8.158.65:37277)
0929 19:03:01.905899 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:01.905912 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:01.905973 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:01.906204 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:01.906205 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:01.906209 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:01.906293 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:01.906295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.907517 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.907518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:01.908962 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:01.908965 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.909712 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.909712 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:01.909784 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:01.909787 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:03:01.909848 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 19:03:01.909883 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":125,"threads_started":1}
I20260929 19:03:01.910053 11064 raft_consensus.cc:2781] T 414f31ccf3cd48debdb07c61252e4463 P 5dce99a6cd474c6abad05507d315d1e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:03:01.910086 11064 raft_consensus.cc:482] T 414f31ccf3cd48debdb07c61252e4463 P 5dce99a6cd474c6abad05507d315d1e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:03:01.910110 11064 raft_consensus.cc:3037] T 414f31ccf3cd48debdb07c61252e4463 P 5dce99a6cd474c6abad05507d315d1e5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:01.910259 10847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414f31ccf3cd48debdb07c61252e4463" candidate_uuid: "5dce99a6cd474c6abad05507d315d1e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1094d0c4c0524fc7b6308499947dddb2" is_pre_election: true
I20260929 19:03:01.910370 10847 raft_consensus.cc:2454] T 414f31ccf3cd48debdb07c61252e4463 P 1094d0c4c0524fc7b6308499947dddb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dce99a6cd474c6abad05507d315d1e5 in term 0.
I20260929 19:03:01.910943 11064 raft_consensus.cc:504] T 414f31ccf3cd48debdb07c61252e4463 P 5dce99a6cd474c6abad05507d315d1e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } }
I20260929 19:03:01.911080 11064 leader_election.cc:288] T 414f31ccf3cd48debdb07c61252e4463 P 5dce99a6cd474c6abad05507d315d1e5 [CANDIDATE]: Term 1 election: Requested vote from peers 1094d0c4c0524fc7b6308499947dddb2 (127.8.158.66:46157), 1b41ba80cad24d3db1d428f488d903d2 (127.8.158.67:38517)
I20260929 19:03:01.911194 10847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414f31ccf3cd48debdb07c61252e4463" candidate_uuid: "5dce99a6cd474c6abad05507d315d1e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1094d0c4c0524fc7b6308499947dddb2"
I20260929 19:03:01.911221 10985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414f31ccf3cd48debdb07c61252e4463" candidate_uuid: "5dce99a6cd474c6abad05507d315d1e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b41ba80cad24d3db1d428f488d903d2"
I20260929 19:03:01.911264 10847 raft_consensus.cc:3037] T 414f31ccf3cd48debdb07c61252e4463 P 1094d0c4c0524fc7b6308499947dddb2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:01.911275 10985 raft_consensus.cc:3037] T 414f31ccf3cd48debdb07c61252e4463 P 1b41ba80cad24d3db1d428f488d903d2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:01.912005 10847 raft_consensus.cc:2454] T 414f31ccf3cd48debdb07c61252e4463 P 1094d0c4c0524fc7b6308499947dddb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dce99a6cd474c6abad05507d315d1e5 in term 1.
I20260929 19:03:01.912040 10985 raft_consensus.cc:2454] T 414f31ccf3cd48debdb07c61252e4463 P 1b41ba80cad24d3db1d428f488d903d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dce99a6cd474c6abad05507d315d1e5 in term 1.
I20260929 19:03:01.912220 10640 leader_election.cc:302] T 414f31ccf3cd48debdb07c61252e4463 P 5dce99a6cd474c6abad05507d315d1e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1094d0c4c0524fc7b6308499947dddb2, 5dce99a6cd474c6abad05507d315d1e5; no voters:
I20260929 19:03:01.912322 11064 raft_consensus.cc:2781] T 414f31ccf3cd48debdb07c61252e4463 P 5dce99a6cd474c6abad05507d315d1e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:03:01.912485 11064 raft_consensus.cc:686] T 414f31ccf3cd48debdb07c61252e4463 P 5dce99a6cd474c6abad05507d315d1e5 [term 1 LEADER]: Becoming Leader. State: Replica: 5dce99a6cd474c6abad05507d315d1e5, State: Running, Role: LEADER
I20260929 19:03:01.912590 11064 consensus_queue.cc:237] T 414f31ccf3cd48debdb07c61252e4463 P 5dce99a6cd474c6abad05507d315d1e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } }
I20260929 19:03:01.913046 11079 tablet_replica.cc:439] T 414f31ccf3cd48debdb07c61252e4463 P 5dce99a6cd474c6abad05507d315d1e5: TxnStatusTablet state changed. Reason: New leader 5dce99a6cd474c6abad05507d315d1e5. Latest consensus state: current_term: 1 leader_uuid: "5dce99a6cd474c6abad05507d315d1e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } } }
I20260929 19:03:01.913130 11079 tablet_replica.cc:442] T 414f31ccf3cd48debdb07c61252e4463 P 5dce99a6cd474c6abad05507d315d1e5: This TxnStatusTablet replica's current role is: LEADER
I20260929 19:03:01.913295 11081 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 19:03:01.913412 11081 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 19:03:01.913404 10550 catalog_manager.cc:5495] T 414f31ccf3cd48debdb07c61252e4463 P 5dce99a6cd474c6abad05507d315d1e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dce99a6cd474c6abad05507d315d1e5 (127.8.158.65). New cstate: current_term: 1 leader_uuid: "5dce99a6cd474c6abad05507d315d1e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } health_report { overall_health: UNKNOWN } } }
I20260929 19:03:01.919989 11055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:01.917001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49864 (local address 127.8.158.65:33741)
0929 19:03:01.917059 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:01.917061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:01.917150 (+ 89us) server_negotiation.cc:407] Connection header received
0929 19:03:01.917276 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:01.917277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:01.917320 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:01.917399 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:03:01.917549 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.918065 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.919149 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.919377 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.919468 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:01.919668 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:03:01.919768 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:03:01.919852 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:01.919908 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 19:03:01.919930 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 19:03:01.920059 11073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:01.916886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:33741 (local address 127.0.0.1:49864)
0929 19:03:01.917072 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:01.917086 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:01.917173 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:01.917404 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:01.917405 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:01.917406 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 19:03:01.917478 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:01.917482 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.918164 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.918165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:01.919022 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:01.919027 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.919473 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.919474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:01.919560 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:01.919576 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 19:03:01.919852 (+ 276us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 19:03:01.919861 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 19:03:01.919906 (+ 45us) client_negotiation.cc:240] Negotiation successful
0929 19:03:01.919973 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260929 19:03:01.921654 10847 raft_consensus.cc:1261] T 414f31ccf3cd48debdb07c61252e4463 P 1094d0c4c0524fc7b6308499947dddb2 [term 1 FOLLOWER]: Refusing update from remote peer 5dce99a6cd474c6abad05507d315d1e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:03:01.921908 10985 raft_consensus.cc:1261] T 414f31ccf3cd48debdb07c61252e4463 P 1b41ba80cad24d3db1d428f488d903d2 [term 1 FOLLOWER]: Refusing update from remote peer 5dce99a6cd474c6abad05507d315d1e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:03:01.921980 11079 consensus_queue.cc:1035] T 414f31ccf3cd48debdb07c61252e4463 P 5dce99a6cd474c6abad05507d315d1e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:03:01.922127 11064 consensus_queue.cc:1035] T 414f31ccf3cd48debdb07c61252e4463 P 5dce99a6cd474c6abad05507d315d1e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:03:01.923310 11062 tablet_replica.cc:439] T 414f31ccf3cd48debdb07c61252e4463 P 1094d0c4c0524fc7b6308499947dddb2: TxnStatusTablet state changed. Reason: New leader 5dce99a6cd474c6abad05507d315d1e5. Latest consensus state: current_term: 1 leader_uuid: "5dce99a6cd474c6abad05507d315d1e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } } }
I20260929 19:03:01.923329 11063 tablet_replica.cc:439] T 414f31ccf3cd48debdb07c61252e4463 P 1b41ba80cad24d3db1d428f488d903d2: TxnStatusTablet state changed. Reason: New leader 5dce99a6cd474c6abad05507d315d1e5. Latest consensus state: current_term: 1 leader_uuid: "5dce99a6cd474c6abad05507d315d1e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } } }
I20260929 19:03:01.923398 11062 tablet_replica.cc:442] T 414f31ccf3cd48debdb07c61252e4463 P 1094d0c4c0524fc7b6308499947dddb2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 19:03:01.923404 11063 tablet_replica.cc:442] T 414f31ccf3cd48debdb07c61252e4463 P 1b41ba80cad24d3db1d428f488d903d2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 19:03:01.924127 11079 tablet_replica.cc:439] T 414f31ccf3cd48debdb07c61252e4463 P 5dce99a6cd474c6abad05507d315d1e5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5dce99a6cd474c6abad05507d315d1e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } } }
I20260929 19:03:01.924219 11079 tablet_replica.cc:442] T 414f31ccf3cd48debdb07c61252e4463 P 5dce99a6cd474c6abad05507d315d1e5: This TxnStatusTablet replica's current role is: LEADER
I20260929 19:03:01.924260 11062 tablet_replica.cc:439] T 414f31ccf3cd48debdb07c61252e4463 P 1094d0c4c0524fc7b6308499947dddb2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5dce99a6cd474c6abad05507d315d1e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } } }
I20260929 19:03:01.924341 11062 tablet_replica.cc:442] T 414f31ccf3cd48debdb07c61252e4463 P 1094d0c4c0524fc7b6308499947dddb2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 19:03:01.924326 11079 tablet_replica.cc:439] T 414f31ccf3cd48debdb07c61252e4463 P 5dce99a6cd474c6abad05507d315d1e5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5dce99a6cd474c6abad05507d315d1e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } } }
I20260929 19:03:01.924381 11079 tablet_replica.cc:442] T 414f31ccf3cd48debdb07c61252e4463 P 5dce99a6cd474c6abad05507d315d1e5: This TxnStatusTablet replica's current role is: LEADER
I20260929 19:03:01.924633 11063 tablet_replica.cc:439] T 414f31ccf3cd48debdb07c61252e4463 P 1b41ba80cad24d3db1d428f488d903d2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5dce99a6cd474c6abad05507d315d1e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } } }
I20260929 19:03:01.924706 11063 tablet_replica.cc:442] T 414f31ccf3cd48debdb07c61252e4463 P 1b41ba80cad24d3db1d428f488d903d2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 19:03:01.924686 11085 mvcc.cc:205] Tried to move back new op lower bound from 7334742351548891136 to 7334742351514669056. Current Snapshot: MvccSnapshot[applied={T|T < 7334742351548891136}]
I20260929 19:03:01.925348 11086 mvcc.cc:205] Tried to move back new op lower bound from 7334742351548891136 to 7334742351514669056. Current Snapshot: MvccSnapshot[applied={T|T < 7334742351548891136}]
I20260929 19:03:01.929615 11053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:01.926860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36674 (local address 127.8.158.67:38517)
0929 19:03:01.926901 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:01.926903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:01.926912 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:03:01.926982 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:01.926983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:01.927000 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:01.927077 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:03:01.927155 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.927903 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.928836 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.929145 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.929213 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:01.929337 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:03:01.929439 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:03:01.929511 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:01.929532 (+ 21us) server_negotiation.cc:299] Negotiation successful
0929 19:03:01.929550 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 19:03:01.934559 11039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:01.931995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:33741 (local address 127.8.158.67:60803)
0929 19:03:01.932070 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:01.932080 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:01.932133 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:01.932252 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:01.932252 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:01.932253 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 19:03:01.932294 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:01.932295 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.933156 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.933157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:01.933712 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:01.933714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.934170 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.934171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:01.934217 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:01.934229 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 19:03:01.934442 (+ 213us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 19:03:01.934444 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 19:03:01.934479 (+ 35us) client_negotiation.cc:240] Negotiation successful
0929 19:03:01.934513 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260929 19:03:01.934576 11055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:01.932052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:60803 (local address 127.8.158.65:33741)
0929 19:03:01.932086 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:01.932088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:01.932115 (+ 27us) server_negotiation.cc:407] Connection header received
0929 19:03:01.932186 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:01.932187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:01.932195 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:01.932251 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:03:01.932336 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.933078 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.933792 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.934094 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.934163 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:01.934336 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:03:01.934368 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:03:01.934420 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:01.934490 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 19:03:01.934507 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 19:03:01.936587 10847 raft_consensus.cc:1261] T bde57a8452e6447a964bc097b88da77c P 1094d0c4c0524fc7b6308499947dddb2 [term 1 FOLLOWER]: Refusing update from remote peer 1b41ba80cad24d3db1d428f488d903d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:03:01.936640 10707 raft_consensus.cc:1261] T bde57a8452e6447a964bc097b88da77c P 5dce99a6cd474c6abad05507d315d1e5 [term 1 FOLLOWER]: Refusing update from remote peer 1b41ba80cad24d3db1d428f488d903d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:03:01.936870 11063 consensus_queue.cc:1035] T bde57a8452e6447a964bc097b88da77c P 1b41ba80cad24d3db1d428f488d903d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:03:01.936981 11066 consensus_queue.cc:1035] T bde57a8452e6447a964bc097b88da77c P 1b41ba80cad24d3db1d428f488d903d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:03:01.945158 11053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:01.942504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:49609 (local address 127.8.158.67:38517)
0929 19:03:01.942537 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:01.942539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:01.942780 (+ 241us) server_negotiation.cc:407] Connection header received
0929 19:03:01.942838 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:01.942839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:01.942850 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:01.942908 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:03:01.943108 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.943567 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.944365 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.944615 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.944694 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:01.944899 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:03:01.944932 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:03:01.944985 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:01.945091 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 19:03:01.945114 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 19:03:01.945210 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:01.942471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.67:38517 (local address 127.8.158.65:49609)
0929 19:03:01.942707 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:01.942718 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:01.942776 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:01.942911 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:01.942912 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:01.942913 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 19:03:01.942975 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:01.942977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.943649 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.943649 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:01.944267 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:01.944268 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:01.944716 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:01.944717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:01.944774 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:01.944789 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 19:03:01.945010 (+ 221us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 19:03:01.945012 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 19:03:01.945079 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 19:03:01.945120 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":77,"threads_started":1}
I20260929 19:03:01.947388 11084 mvcc.cc:205] Tried to move back new op lower bound from 7334742351610560512 to 7334742351125159936. Current Snapshot: MvccSnapshot[applied={T|T < 7334742351610560512}]
I20260929 19:03:01.954464 10550 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:03:01.954574 10550 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:03:01.955210 10550 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 06a804e1e1b34f37b46d8763e45660ad: Sending DeleteTablet for 3 replicas of tablet bde57a8452e6447a964bc097b88da77c
I20260929 19:03:01.955542 10687 tablet_service.cc:1514] Processing DeleteTablet for tablet bde57a8452e6447a964bc097b88da77c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:01 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:49858
I20260929 19:03:01.955544 10827 tablet_service.cc:1514] Processing DeleteTablet for tablet bde57a8452e6447a964bc097b88da77c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:01 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:45686
I20260929 19:03:01.955785 10967 tablet_service.cc:1514] Processing DeleteTablet for tablet bde57a8452e6447a964bc097b88da77c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:01 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:36660
I20260929 19:03:01.956014 11106 tablet_replica.cc:330] T bde57a8452e6447a964bc097b88da77c P 1b41ba80cad24d3db1d428f488d903d2: stopping tablet replica
I20260929 19:03:01.956148 11106 raft_consensus.cc:2229] T bde57a8452e6447a964bc097b88da77c P 1b41ba80cad24d3db1d428f488d903d2 [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:03:01.956223 10550 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:03:01.956377 10550 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:03:01.956306 11106 raft_consensus.cc:2258] T bde57a8452e6447a964bc097b88da77c P 1b41ba80cad24d3db1d428f488d903d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:03:01.956928 11106 ts_tablet_manager.cc:1905] T bde57a8452e6447a964bc097b88da77c P 1b41ba80cad24d3db1d428f488d903d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:03:01.957547 10687 tablet_service.cc:1467] Processing CreateTablet for tablet 3c1fd6b86fb74ee485de1ce1b97aa042 (DEFAULT_TABLE table=test-table [id=26d6b9824ddc4fb487f67a356c860afa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:01.957661 10687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c1fd6b86fb74ee485de1ce1b97aa042. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:01.958860 11106 ts_tablet_manager.cc:1918] T bde57a8452e6447a964bc097b88da77c P 1b41ba80cad24d3db1d428f488d903d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 19:03:01.959007 11058 tablet_bootstrap.cc:492] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 5dce99a6cd474c6abad05507d315d1e5: Bootstrap starting.
I20260929 19:03:01.959393 11058 tablet_bootstrap.cc:654] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 5dce99a6cd474c6abad05507d315d1e5: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:01.959410 10827 tablet_service.cc:1467] Processing CreateTablet for tablet 3c1fd6b86fb74ee485de1ce1b97aa042 (DEFAULT_TABLE table=test-table [id=26d6b9824ddc4fb487f67a356c860afa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:01.959517 10827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c1fd6b86fb74ee485de1ce1b97aa042. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:01.959223 10967 tablet_service.cc:1467] Processing CreateTablet for tablet 3c1fd6b86fb74ee485de1ce1b97aa042 (DEFAULT_TABLE table=test-table [id=26d6b9824ddc4fb487f67a356c860afa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:01.960079 10967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c1fd6b86fb74ee485de1ce1b97aa042. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:01.960808 11057 tablet_bootstrap.cc:492] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 1094d0c4c0524fc7b6308499947dddb2: Bootstrap starting.
I20260929 19:03:01.961148 11057 tablet_bootstrap.cc:654] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 1094d0c4c0524fc7b6308499947dddb2: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:01.961277 11056 tablet_bootstrap.cc:492] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 1b41ba80cad24d3db1d428f488d903d2: Bootstrap starting.
I20260929 19:03:01.961653 11056 tablet_bootstrap.cc:654] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 1b41ba80cad24d3db1d428f488d903d2: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:01.962149 11106 log.cc:1196] T bde57a8452e6447a964bc097b88da77c P 1b41ba80cad24d3db1d428f488d903d2: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-2/wal/wals/bde57a8452e6447a964bc097b88da77c
I20260929 19:03:01.962190 11057 tablet_bootstrap.cc:492] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 1094d0c4c0524fc7b6308499947dddb2: No bootstrap required, opened a new log
I20260929 19:03:01.962235 11057 ts_tablet_manager.cc:1397] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 1094d0c4c0524fc7b6308499947dddb2: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 19:03:01.962416 11057 raft_consensus.cc:348] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 1094d0c4c0524fc7b6308499947dddb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } }
I20260929 19:03:01.962503 11057 raft_consensus.cc:374] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 1094d0c4c0524fc7b6308499947dddb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:01.962532 11057 raft_consensus.cc:729] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 1094d0c4c0524fc7b6308499947dddb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1094d0c4c0524fc7b6308499947dddb2, State: Initialized, Role: FOLLOWER
I20260929 19:03:01.962574 11057 consensus_queue.cc:260] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 1094d0c4c0524fc7b6308499947dddb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } }
I20260929 19:03:01.962670 11057 ts_tablet_manager.cc:1428] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 1094d0c4c0524fc7b6308499947dddb2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 19:03:01.962805 11106 ts_tablet_manager.cc:1939] T bde57a8452e6447a964bc097b88da77c P 1b41ba80cad24d3db1d428f488d903d2: Deleting consensus metadata
I20260929 19:03:01.963213 10534 catalog_manager.cc:4841] TS 1b41ba80cad24d3db1d428f488d903d2 (127.8.158.67:38517): tablet bde57a8452e6447a964bc097b88da77c (table test-table [id=5f3d6556a184479d9ba59213cebc17d9]) successfully deleted
I20260929 19:03:01.963186 11104 tablet_replica.cc:330] T bde57a8452e6447a964bc097b88da77c P 1094d0c4c0524fc7b6308499947dddb2: stopping tablet replica
I20260929 19:03:01.963569 11104 raft_consensus.cc:2229] T bde57a8452e6447a964bc097b88da77c P 1094d0c4c0524fc7b6308499947dddb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:03:01.963629 11104 raft_consensus.cc:2258] T bde57a8452e6447a964bc097b88da77c P 1094d0c4c0524fc7b6308499947dddb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:03:01.963696 11056 tablet_bootstrap.cc:492] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 1b41ba80cad24d3db1d428f488d903d2: No bootstrap required, opened a new log
I20260929 19:03:01.963737 11056 ts_tablet_manager.cc:1397] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 1b41ba80cad24d3db1d428f488d903d2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 19:03:01.963773 11105 tablet_replica.cc:330] T bde57a8452e6447a964bc097b88da77c P 5dce99a6cd474c6abad05507d315d1e5: stopping tablet replica
I20260929 19:03:01.963863 11105 raft_consensus.cc:2229] T bde57a8452e6447a964bc097b88da77c P 5dce99a6cd474c6abad05507d315d1e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:03:01.963905 11105 raft_consensus.cc:2258] T bde57a8452e6447a964bc097b88da77c P 5dce99a6cd474c6abad05507d315d1e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:03:01.963861 11056 raft_consensus.cc:348] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 1b41ba80cad24d3db1d428f488d903d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } }
I20260929 19:03:01.963934 11056 raft_consensus.cc:374] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 1b41ba80cad24d3db1d428f488d903d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:01.963958 11056 raft_consensus.cc:729] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 1b41ba80cad24d3db1d428f488d903d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b41ba80cad24d3db1d428f488d903d2, State: Initialized, Role: FOLLOWER
I20260929 19:03:01.964000 11056 consensus_queue.cc:260] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 1b41ba80cad24d3db1d428f488d903d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } }
I20260929 19:03:01.964087 11056 ts_tablet_manager.cc:1428] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 1b41ba80cad24d3db1d428f488d903d2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:01.964464 11104 ts_tablet_manager.cc:1905] T bde57a8452e6447a964bc097b88da77c P 1094d0c4c0524fc7b6308499947dddb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:03:01.964715 11105 ts_tablet_manager.cc:1905] T bde57a8452e6447a964bc097b88da77c P 5dce99a6cd474c6abad05507d315d1e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:03:01.966245 11104 ts_tablet_manager.cc:1918] T bde57a8452e6447a964bc097b88da77c P 1094d0c4c0524fc7b6308499947dddb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 19:03:01.966364 11104 log.cc:1196] T bde57a8452e6447a964bc097b88da77c P 1094d0c4c0524fc7b6308499947dddb2: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-1/wal/wals/bde57a8452e6447a964bc097b88da77c
I20260929 19:03:01.966514 11058 tablet_bootstrap.cc:492] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 5dce99a6cd474c6abad05507d315d1e5: No bootstrap required, opened a new log
I20260929 19:03:01.966555 11058 ts_tablet_manager.cc:1397] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 5dce99a6cd474c6abad05507d315d1e5: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20260929 19:03:01.966631 11104 ts_tablet_manager.cc:1939] T bde57a8452e6447a964bc097b88da77c P 1094d0c4c0524fc7b6308499947dddb2: Deleting consensus metadata
I20260929 19:03:01.966684 11058 raft_consensus.cc:348] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 5dce99a6cd474c6abad05507d315d1e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } }
I20260929 19:03:01.966775 11058 raft_consensus.cc:374] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 5dce99a6cd474c6abad05507d315d1e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:01.966795 11058 raft_consensus.cc:729] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 5dce99a6cd474c6abad05507d315d1e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dce99a6cd474c6abad05507d315d1e5, State: Initialized, Role: FOLLOWER
I20260929 19:03:01.966840 11058 consensus_queue.cc:260] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 5dce99a6cd474c6abad05507d315d1e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } }
I20260929 19:03:01.966943 11058 ts_tablet_manager.cc:1428] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 5dce99a6cd474c6abad05507d315d1e5: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260929 19:03:01.967115 10536 catalog_manager.cc:4841] TS 1094d0c4c0524fc7b6308499947dddb2 (127.8.158.66:46157): tablet bde57a8452e6447a964bc097b88da77c (table test-table [id=5f3d6556a184479d9ba59213cebc17d9]) successfully deleted
I20260929 19:03:01.967273 11105 ts_tablet_manager.cc:1918] T bde57a8452e6447a964bc097b88da77c P 5dce99a6cd474c6abad05507d315d1e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 19:03:01.967510 11105 log.cc:1196] T bde57a8452e6447a964bc097b88da77c P 5dce99a6cd474c6abad05507d315d1e5: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-0/wal/wals/bde57a8452e6447a964bc097b88da77c
I20260929 19:03:01.967842 11105 ts_tablet_manager.cc:1939] T bde57a8452e6447a964bc097b88da77c P 5dce99a6cd474c6abad05507d315d1e5: Deleting consensus metadata
I20260929 19:03:01.968209 10535 catalog_manager.cc:4841] TS 5dce99a6cd474c6abad05507d315d1e5 (127.8.158.65:33741): tablet bde57a8452e6447a964bc097b88da77c (table test-table [id=5f3d6556a184479d9ba59213cebc17d9]) successfully deleted
I20260929 19:03:02.307355 11097 raft_consensus.cc:482] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 5dce99a6cd474c6abad05507d315d1e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:03:02.307482 11097 raft_consensus.cc:504] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 5dce99a6cd474c6abad05507d315d1e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } }
I20260929 19:03:02.307659 11097 leader_election.cc:288] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 5dce99a6cd474c6abad05507d315d1e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1094d0c4c0524fc7b6308499947dddb2 (127.8.158.66:46157), 1b41ba80cad24d3db1d428f488d903d2 (127.8.158.67:38517)
I20260929 19:03:02.307912 10985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1fd6b86fb74ee485de1ce1b97aa042" candidate_uuid: "5dce99a6cd474c6abad05507d315d1e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b41ba80cad24d3db1d428f488d903d2" is_pre_election: true
I20260929 19:03:02.307900 10847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1fd6b86fb74ee485de1ce1b97aa042" candidate_uuid: "5dce99a6cd474c6abad05507d315d1e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1094d0c4c0524fc7b6308499947dddb2" is_pre_election: true
I20260929 19:03:02.308005 10847 raft_consensus.cc:2454] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 1094d0c4c0524fc7b6308499947dddb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dce99a6cd474c6abad05507d315d1e5 in term 0.
I20260929 19:03:02.308005 10985 raft_consensus.cc:2454] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 1b41ba80cad24d3db1d428f488d903d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dce99a6cd474c6abad05507d315d1e5 in term 0.
I20260929 19:03:02.308166 10640 leader_election.cc:302] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 5dce99a6cd474c6abad05507d315d1e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1094d0c4c0524fc7b6308499947dddb2, 5dce99a6cd474c6abad05507d315d1e5; no voters:
I20260929 19:03:02.308279 11097 raft_consensus.cc:2781] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 5dce99a6cd474c6abad05507d315d1e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:03:02.308320 11097 raft_consensus.cc:482] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 5dce99a6cd474c6abad05507d315d1e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:03:02.308341 11097 raft_consensus.cc:3037] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 5dce99a6cd474c6abad05507d315d1e5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:02.309048 11097 raft_consensus.cc:504] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 5dce99a6cd474c6abad05507d315d1e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } }
I20260929 19:03:02.309181 11097 leader_election.cc:288] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 5dce99a6cd474c6abad05507d315d1e5 [CANDIDATE]: Term 1 election: Requested vote from peers 1094d0c4c0524fc7b6308499947dddb2 (127.8.158.66:46157), 1b41ba80cad24d3db1d428f488d903d2 (127.8.158.67:38517)
I20260929 19:03:02.309345 10985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1fd6b86fb74ee485de1ce1b97aa042" candidate_uuid: "5dce99a6cd474c6abad05507d315d1e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b41ba80cad24d3db1d428f488d903d2"
I20260929 19:03:02.309348 10847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1fd6b86fb74ee485de1ce1b97aa042" candidate_uuid: "5dce99a6cd474c6abad05507d315d1e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1094d0c4c0524fc7b6308499947dddb2"
I20260929 19:03:02.309418 10985 raft_consensus.cc:3037] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 1b41ba80cad24d3db1d428f488d903d2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:02.309418 10847 raft_consensus.cc:3037] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 1094d0c4c0524fc7b6308499947dddb2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:02.310120 10985 raft_consensus.cc:2454] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 1b41ba80cad24d3db1d428f488d903d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dce99a6cd474c6abad05507d315d1e5 in term 1.
I20260929 19:03:02.310132 10847 raft_consensus.cc:2454] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 1094d0c4c0524fc7b6308499947dddb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dce99a6cd474c6abad05507d315d1e5 in term 1.
I20260929 19:03:02.310348 10638 leader_election.cc:302] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 5dce99a6cd474c6abad05507d315d1e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b41ba80cad24d3db1d428f488d903d2, 5dce99a6cd474c6abad05507d315d1e5; no voters:
I20260929 19:03:02.310472 11097 raft_consensus.cc:2781] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 5dce99a6cd474c6abad05507d315d1e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:03:02.310551 11097 raft_consensus.cc:686] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 5dce99a6cd474c6abad05507d315d1e5 [term 1 LEADER]: Becoming Leader. State: Replica: 5dce99a6cd474c6abad05507d315d1e5, State: Running, Role: LEADER
I20260929 19:03:02.310640 11097 consensus_queue.cc:237] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 5dce99a6cd474c6abad05507d315d1e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } }
I20260929 19:03:02.311170 10550 catalog_manager.cc:5495] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 5dce99a6cd474c6abad05507d315d1e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dce99a6cd474c6abad05507d315d1e5 (127.8.158.65). New cstate: current_term: 1 leader_uuid: "5dce99a6cd474c6abad05507d315d1e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } health_report { overall_health: UNKNOWN } } }
I20260929 19:03:02.318517 11055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:02.316544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49876 (local address 127.8.158.65:33741)
0929 19:03:02.316594 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:02.316595 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:02.316602 (+ 7us) server_negotiation.cc:407] Connection header received
0929 19:03:02.316619 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:02.316620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:02.316628 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:02.316672 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:03:02.316753 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:02.317221 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:02.317945 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:02.318118 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:02.318173 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:02.318284 (+ 111us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:03:02.318307 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:03:02.318406 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:02.318454 (+ 48us) server_negotiation.cc:299] Negotiation successful
0929 19:03:02.318468 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 19:03:02.322157 11055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:02.320340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:32871 (local address 127.8.158.65:33741)
0929 19:03:02.320370 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:02.320372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:02.320380 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:03:02.320433 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:02.320433 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:02.320440 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:02.320476 (+ 36us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:03:02.320578 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:02.320995 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:02.321637 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:02.321796 (+ 159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:02.321842 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:02.321968 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:03:02.321988 (+ 20us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:03:02.322025 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:02.322074 (+ 49us) server_negotiation.cc:299] Negotiation successful
0929 19:03:02.322091 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":14}
I20260929 19:03:02.322201 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:02.320277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:33741 (local address 127.8.158.65:32871)
0929 19:03:02.320331 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:02.320341 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:02.320378 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:02.320490 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:02.320491 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:02.320492 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 19:03:02.320519 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:02.320519 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:02.321062 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:02.321063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:02.321570 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:02.321571 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:02.321858 (+ 287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:02.321866 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:02.321902 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:02.321907 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 19:03:02.322040 (+ 133us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 19:03:02.322042 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 19:03:02.322074 (+ 32us) client_negotiation.cc:240] Negotiation successful
0929 19:03:02.322107 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":1}
I20260929 19:03:02.323920 10985 raft_consensus.cc:1261] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 1b41ba80cad24d3db1d428f488d903d2 [term 1 FOLLOWER]: Refusing update from remote peer 5dce99a6cd474c6abad05507d315d1e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:03:02.323931 10847 raft_consensus.cc:1261] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 1094d0c4c0524fc7b6308499947dddb2 [term 1 FOLLOWER]: Refusing update from remote peer 5dce99a6cd474c6abad05507d315d1e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:03:02.324115 11079 consensus_queue.cc:1035] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 5dce99a6cd474c6abad05507d315d1e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:03:02.324195 11097 consensus_queue.cc:1035] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 5dce99a6cd474c6abad05507d315d1e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 19:03:02.328016 11084 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 19:03:03.846230 10550 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:03:03.846369 10550 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:03:03.847177 10550 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 06a804e1e1b34f37b46d8763e45660ad: Sending DeleteTablet for 3 replicas of tablet 3c1fd6b86fb74ee485de1ce1b97aa042
I20260929 19:03:03.847437 10827 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c1fd6b86fb74ee485de1ce1b97aa042 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:03 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:45686
I20260929 19:03:03.847440 10687 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c1fd6b86fb74ee485de1ce1b97aa042 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:03 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:49858
I20260929 19:03:03.847492 10967 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c1fd6b86fb74ee485de1ce1b97aa042 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:03 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:36660
I20260929 19:03:03.847687 11136 tablet_replica.cc:330] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 1b41ba80cad24d3db1d428f488d903d2: stopping tablet replica
I20260929 19:03:03.847698 11134 tablet_replica.cc:330] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 1094d0c4c0524fc7b6308499947dddb2: stopping tablet replica
I20260929 19:03:03.847764 11134 raft_consensus.cc:2229] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 1094d0c4c0524fc7b6308499947dddb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:03:03.847764 11136 raft_consensus.cc:2229] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 1b41ba80cad24d3db1d428f488d903d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:03:03.847812 11136 raft_consensus.cc:2258] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 1b41ba80cad24d3db1d428f488d903d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:03:03.847811 11134 raft_consensus.cc:2258] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 1094d0c4c0524fc7b6308499947dddb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:03:03.848003 11135 tablet_replica.cc:330] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 5dce99a6cd474c6abad05507d315d1e5: stopping tablet replica
I20260929 19:03:03.848079 11136 ts_tablet_manager.cc:1905] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 1b41ba80cad24d3db1d428f488d903d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:03:03.848080 11134 ts_tablet_manager.cc:1905] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 1094d0c4c0524fc7b6308499947dddb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:03:03.848124 11135 raft_consensus.cc:2229] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 5dce99a6cd474c6abad05507d315d1e5 [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:03:03.848282 11135 raft_consensus.cc:2258] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 5dce99a6cd474c6abad05507d315d1e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:03:03.848546 11135 ts_tablet_manager.cc:1905] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 5dce99a6cd474c6abad05507d315d1e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:03:03.849861 11134 ts_tablet_manager.cc:1918] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 1094d0c4c0524fc7b6308499947dddb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 19:03:03.849861 11136 ts_tablet_manager.cc:1918] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 1b41ba80cad24d3db1d428f488d903d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 19:03:03.849924 11136 log.cc:1196] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 1b41ba80cad24d3db1d428f488d903d2: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-2/wal/wals/3c1fd6b86fb74ee485de1ce1b97aa042
I20260929 19:03:03.850004 11134 log.cc:1196] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 1094d0c4c0524fc7b6308499947dddb2: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-1/wal/wals/3c1fd6b86fb74ee485de1ce1b97aa042
I20260929 19:03:03.850210 11136 ts_tablet_manager.cc:1939] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 1b41ba80cad24d3db1d428f488d903d2: Deleting consensus metadata
I20260929 19:03:03.850415 11134 ts_tablet_manager.cc:1939] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 1094d0c4c0524fc7b6308499947dddb2: Deleting consensus metadata
I20260929 19:03:03.850461 11135 ts_tablet_manager.cc:1918] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 5dce99a6cd474c6abad05507d315d1e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 19:03:03.850503 11135 log.cc:1196] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 5dce99a6cd474c6abad05507d315d1e5: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-0/wal/wals/3c1fd6b86fb74ee485de1ce1b97aa042
I20260929 19:03:03.850697 10534 catalog_manager.cc:4841] TS 1b41ba80cad24d3db1d428f488d903d2 (127.8.158.67:38517): tablet 3c1fd6b86fb74ee485de1ce1b97aa042 (table test-table [id=26d6b9824ddc4fb487f67a356c860afa]) successfully deleted
I20260929 19:03:03.850725 11135 ts_tablet_manager.cc:1939] T 3c1fd6b86fb74ee485de1ce1b97aa042 P 5dce99a6cd474c6abad05507d315d1e5: Deleting consensus metadata
I20260929 19:03:03.850841 10536 catalog_manager.cc:4841] TS 1094d0c4c0524fc7b6308499947dddb2 (127.8.158.66:46157): tablet 3c1fd6b86fb74ee485de1ce1b97aa042 (table test-table [id=26d6b9824ddc4fb487f67a356c860afa]) successfully deleted
I20260929 19:03:03.851006 10535 catalog_manager.cc:4841] TS 5dce99a6cd474c6abad05507d315d1e5 (127.8.158.65:33741): tablet 3c1fd6b86fb74ee485de1ce1b97aa042 (table test-table [id=26d6b9824ddc4fb487f67a356c860afa]) successfully deleted
Sep 29 19:03:03 dist-test-slave-nmsz krb5kdc[10505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708583, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 19:03:03 dist-test-slave-nmsz krb5kdc[10505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708583, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:03:03.878428 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:03.868620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35888 (local address 127.8.158.126:33825)
0929 19:03:03.868867 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:03.868869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:03.868879 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:03:03.868904 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:03.868905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:03.868918 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:03.869001 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:03.869721 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:03.870540 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:03.871703 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:03.871897 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:03.871962 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:03.875038 (+ 3076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:03.875039 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:03.875040 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:03.875046 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:03.876672 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:03.877077 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:03.877078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:03.877078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:03.877170 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:03.877654 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:03.877655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:03.877655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:03.877807 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:03.877900 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:03.878207 (+ 307us) server_negotiation.cc:299] Negotiation successful
0929 19:03:03.878290 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":69,"threads_started":1}
I20260929 19:03:03.879987 10550 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35888:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:03:03.880165 10550 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:03:03.881621 10687 tablet_service.cc:1467] Processing CreateTablet for tablet 18372107f12f454bad8ef2177b0269dd (DEFAULT_TABLE table=test-table [id=0b5a972b6a134244937ab2e9b8d01f6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:03.881625 10827 tablet_service.cc:1467] Processing CreateTablet for tablet 18372107f12f454bad8ef2177b0269dd (DEFAULT_TABLE table=test-table [id=0b5a972b6a134244937ab2e9b8d01f6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:03.881664 10967 tablet_service.cc:1467] Processing CreateTablet for tablet 18372107f12f454bad8ef2177b0269dd (DEFAULT_TABLE table=test-table [id=0b5a972b6a134244937ab2e9b8d01f6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:03.881743 10827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18372107f12f454bad8ef2177b0269dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:03.881744 10687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18372107f12f454bad8ef2177b0269dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:03.881788 10967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18372107f12f454bad8ef2177b0269dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:03.883121 11145 tablet_bootstrap.cc:492] T 18372107f12f454bad8ef2177b0269dd P 1094d0c4c0524fc7b6308499947dddb2: Bootstrap starting.
I20260929 19:03:03.883289 11147 tablet_bootstrap.cc:492] T 18372107f12f454bad8ef2177b0269dd P 1b41ba80cad24d3db1d428f488d903d2: Bootstrap starting.
I20260929 19:03:03.883628 11147 tablet_bootstrap.cc:654] T 18372107f12f454bad8ef2177b0269dd P 1b41ba80cad24d3db1d428f488d903d2: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:03.883632 11145 tablet_bootstrap.cc:654] T 18372107f12f454bad8ef2177b0269dd P 1094d0c4c0524fc7b6308499947dddb2: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:03.883927 11146 tablet_bootstrap.cc:492] T 18372107f12f454bad8ef2177b0269dd P 5dce99a6cd474c6abad05507d315d1e5: Bootstrap starting.
I20260929 19:03:03.884223 11147 tablet_bootstrap.cc:492] T 18372107f12f454bad8ef2177b0269dd P 1b41ba80cad24d3db1d428f488d903d2: No bootstrap required, opened a new log
I20260929 19:03:03.884245 11146 tablet_bootstrap.cc:654] T 18372107f12f454bad8ef2177b0269dd P 5dce99a6cd474c6abad05507d315d1e5: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:03.884265 11147 ts_tablet_manager.cc:1397] T 18372107f12f454bad8ef2177b0269dd P 1b41ba80cad24d3db1d428f488d903d2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:03.884362 11145 tablet_bootstrap.cc:492] T 18372107f12f454bad8ef2177b0269dd P 1094d0c4c0524fc7b6308499947dddb2: No bootstrap required, opened a new log
I20260929 19:03:03.884367 11147 raft_consensus.cc:348] T 18372107f12f454bad8ef2177b0269dd P 1b41ba80cad24d3db1d428f488d903d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } }
I20260929 19:03:03.884413 11147 raft_consensus.cc:374] T 18372107f12f454bad8ef2177b0269dd P 1b41ba80cad24d3db1d428f488d903d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:03.884430 11147 raft_consensus.cc:729] T 18372107f12f454bad8ef2177b0269dd P 1b41ba80cad24d3db1d428f488d903d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b41ba80cad24d3db1d428f488d903d2, State: Initialized, Role: FOLLOWER
I20260929 19:03:03.884488 11147 consensus_queue.cc:260] T 18372107f12f454bad8ef2177b0269dd P 1b41ba80cad24d3db1d428f488d903d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } }
I20260929 19:03:03.884521 11145 ts_tablet_manager.cc:1397] T 18372107f12f454bad8ef2177b0269dd P 1094d0c4c0524fc7b6308499947dddb2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:03.884655 11147 ts_tablet_manager.cc:1428] T 18372107f12f454bad8ef2177b0269dd P 1b41ba80cad24d3db1d428f488d903d2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 19:03:03.884756 11145 raft_consensus.cc:348] T 18372107f12f454bad8ef2177b0269dd P 1094d0c4c0524fc7b6308499947dddb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } }
I20260929 19:03:03.884811 11146 tablet_bootstrap.cc:492] T 18372107f12f454bad8ef2177b0269dd P 5dce99a6cd474c6abad05507d315d1e5: No bootstrap required, opened a new log
I20260929 19:03:03.884855 11146 ts_tablet_manager.cc:1397] T 18372107f12f454bad8ef2177b0269dd P 5dce99a6cd474c6abad05507d315d1e5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:03.884865 11145 raft_consensus.cc:374] T 18372107f12f454bad8ef2177b0269dd P 1094d0c4c0524fc7b6308499947dddb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:03.884923 11145 raft_consensus.cc:729] T 18372107f12f454bad8ef2177b0269dd P 1094d0c4c0524fc7b6308499947dddb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1094d0c4c0524fc7b6308499947dddb2, State: Initialized, Role: FOLLOWER
I20260929 19:03:03.884987 11145 consensus_queue.cc:260] T 18372107f12f454bad8ef2177b0269dd P 1094d0c4c0524fc7b6308499947dddb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } }
I20260929 19:03:03.884972 11146 raft_consensus.cc:348] T 18372107f12f454bad8ef2177b0269dd P 5dce99a6cd474c6abad05507d315d1e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } }
I20260929 19:03:03.885083 11146 raft_consensus.cc:374] T 18372107f12f454bad8ef2177b0269dd P 5dce99a6cd474c6abad05507d315d1e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:03.885113 11146 raft_consensus.cc:729] T 18372107f12f454bad8ef2177b0269dd P 5dce99a6cd474c6abad05507d315d1e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dce99a6cd474c6abad05507d315d1e5, State: Initialized, Role: FOLLOWER
I20260929 19:03:03.885178 11146 consensus_queue.cc:260] T 18372107f12f454bad8ef2177b0269dd P 5dce99a6cd474c6abad05507d315d1e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } }
I20260929 19:03:03.885270 11145 ts_tablet_manager.cc:1428] T 18372107f12f454bad8ef2177b0269dd P 1094d0c4c0524fc7b6308499947dddb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 19:03:03.885442 11146 ts_tablet_manager.cc:1428] T 18372107f12f454bad8ef2177b0269dd P 5dce99a6cd474c6abad05507d315d1e5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:03.896996 11151 raft_consensus.cc:482] T 18372107f12f454bad8ef2177b0269dd P 1b41ba80cad24d3db1d428f488d903d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:03:03.897087 11151 raft_consensus.cc:504] T 18372107f12f454bad8ef2177b0269dd P 1b41ba80cad24d3db1d428f488d903d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } }
I20260929 19:03:03.897297 11151 leader_election.cc:288] T 18372107f12f454bad8ef2177b0269dd P 1b41ba80cad24d3db1d428f488d903d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dce99a6cd474c6abad05507d315d1e5 (127.8.158.65:33741), 1094d0c4c0524fc7b6308499947dddb2 (127.8.158.66:46157)
I20260929 19:03:03.897377 10847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18372107f12f454bad8ef2177b0269dd" candidate_uuid: "1b41ba80cad24d3db1d428f488d903d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1094d0c4c0524fc7b6308499947dddb2" is_pre_election: true
I20260929 19:03:03.897392 10707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18372107f12f454bad8ef2177b0269dd" candidate_uuid: "1b41ba80cad24d3db1d428f488d903d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dce99a6cd474c6abad05507d315d1e5" is_pre_election: true
I20260929 19:03:03.897449 10847 raft_consensus.cc:2454] T 18372107f12f454bad8ef2177b0269dd P 1094d0c4c0524fc7b6308499947dddb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b41ba80cad24d3db1d428f488d903d2 in term 0.
I20260929 19:03:03.897490 10707 raft_consensus.cc:2454] T 18372107f12f454bad8ef2177b0269dd P 5dce99a6cd474c6abad05507d315d1e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b41ba80cad24d3db1d428f488d903d2 in term 0.
I20260929 19:03:03.897588 10920 leader_election.cc:302] T 18372107f12f454bad8ef2177b0269dd P 1b41ba80cad24d3db1d428f488d903d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1094d0c4c0524fc7b6308499947dddb2, 1b41ba80cad24d3db1d428f488d903d2; no voters:
I20260929 19:03:03.897745 11153 raft_consensus.cc:2781] T 18372107f12f454bad8ef2177b0269dd P 1b41ba80cad24d3db1d428f488d903d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:03:03.897809 11153 raft_consensus.cc:482] T 18372107f12f454bad8ef2177b0269dd P 1b41ba80cad24d3db1d428f488d903d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:03:03.897835 11153 raft_consensus.cc:3037] T 18372107f12f454bad8ef2177b0269dd P 1b41ba80cad24d3db1d428f488d903d2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:03.898769 11153 raft_consensus.cc:504] T 18372107f12f454bad8ef2177b0269dd P 1b41ba80cad24d3db1d428f488d903d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } }
I20260929 19:03:03.898908 11153 leader_election.cc:288] T 18372107f12f454bad8ef2177b0269dd P 1b41ba80cad24d3db1d428f488d903d2 [CANDIDATE]: Term 1 election: Requested vote from peers 5dce99a6cd474c6abad05507d315d1e5 (127.8.158.65:33741), 1094d0c4c0524fc7b6308499947dddb2 (127.8.158.66:46157)
I20260929 19:03:03.899035 10707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18372107f12f454bad8ef2177b0269dd" candidate_uuid: "1b41ba80cad24d3db1d428f488d903d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dce99a6cd474c6abad05507d315d1e5"
I20260929 19:03:03.899112 10707 raft_consensus.cc:3037] T 18372107f12f454bad8ef2177b0269dd P 5dce99a6cd474c6abad05507d315d1e5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:03.899928 10707 raft_consensus.cc:2454] T 18372107f12f454bad8ef2177b0269dd P 5dce99a6cd474c6abad05507d315d1e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b41ba80cad24d3db1d428f488d903d2 in term 1.
I20260929 19:03:03.900079 10919 leader_election.cc:302] T 18372107f12f454bad8ef2177b0269dd P 1b41ba80cad24d3db1d428f488d903d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b41ba80cad24d3db1d428f488d903d2, 5dce99a6cd474c6abad05507d315d1e5; no voters:
I20260929 19:03:03.900108 10847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18372107f12f454bad8ef2177b0269dd" candidate_uuid: "1b41ba80cad24d3db1d428f488d903d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1094d0c4c0524fc7b6308499947dddb2"
I20260929 19:03:03.900173 10847 raft_consensus.cc:3037] T 18372107f12f454bad8ef2177b0269dd P 1094d0c4c0524fc7b6308499947dddb2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:03.900941 10847 raft_consensus.cc:2454] T 18372107f12f454bad8ef2177b0269dd P 1094d0c4c0524fc7b6308499947dddb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b41ba80cad24d3db1d428f488d903d2 in term 1.
I20260929 19:03:03.901044 11153 raft_consensus.cc:2781] T 18372107f12f454bad8ef2177b0269dd P 1b41ba80cad24d3db1d428f488d903d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:03:03.901105 11153 raft_consensus.cc:686] T 18372107f12f454bad8ef2177b0269dd P 1b41ba80cad24d3db1d428f488d903d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1b41ba80cad24d3db1d428f488d903d2, State: Running, Role: LEADER
I20260929 19:03:03.901163 11153 consensus_queue.cc:237] T 18372107f12f454bad8ef2177b0269dd P 1b41ba80cad24d3db1d428f488d903d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } }
I20260929 19:03:03.901691 10547 catalog_manager.cc:5495] T 18372107f12f454bad8ef2177b0269dd P 1b41ba80cad24d3db1d428f488d903d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b41ba80cad24d3db1d428f488d903d2 (127.8.158.67). New cstate: current_term: 1 leader_uuid: "1b41ba80cad24d3db1d428f488d903d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } health_report { overall_health: HEALTHY } } }
I20260929 19:03:03.909365 11156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:03.906936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36676 (local address 127.8.158.67:38517)
0929 19:03:03.907077 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:03.907079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:03.907085 (+ 6us) server_negotiation.cc:407] Connection header received
0929 19:03:03.907104 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:03.907105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:03.907121 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:03.907253 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:03:03.907380 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:03.907926 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:03.908675 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:03.908884 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:03.908960 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:03.909109 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:03:03.909139 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:03:03.909208 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:03.909266 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 19:03:03.909286 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
I20260929 19:03:03.911334 10707 raft_consensus.cc:1261] T 18372107f12f454bad8ef2177b0269dd P 5dce99a6cd474c6abad05507d315d1e5 [term 1 FOLLOWER]: Refusing update from remote peer 1b41ba80cad24d3db1d428f488d903d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:03:03.911406 10847 raft_consensus.cc:1261] T 18372107f12f454bad8ef2177b0269dd P 1094d0c4c0524fc7b6308499947dddb2 [term 1 FOLLOWER]: Refusing update from remote peer 1b41ba80cad24d3db1d428f488d903d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:03:03.911568 11153 consensus_queue.cc:1035] T 18372107f12f454bad8ef2177b0269dd P 1b41ba80cad24d3db1d428f488d903d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:03:03.911666 11151 consensus_queue.cc:1035] T 18372107f12f454bad8ef2177b0269dd P 1b41ba80cad24d3db1d428f488d903d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:03:03.920749 10550 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:03:03.920836 10550 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:03:03.921396 10550 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 06a804e1e1b34f37b46d8763e45660ad: Sending DeleteTablet for 3 replicas of tablet 18372107f12f454bad8ef2177b0269dd
I20260929 19:03:03.921628 10687 tablet_service.cc:1514] Processing DeleteTablet for tablet 18372107f12f454bad8ef2177b0269dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:03 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:49858
I20260929 19:03:03.921625 10827 tablet_service.cc:1514] Processing DeleteTablet for tablet 18372107f12f454bad8ef2177b0269dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:03 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:45686
I20260929 19:03:03.921734 11135 tablet_replica.cc:330] T 18372107f12f454bad8ef2177b0269dd P 5dce99a6cd474c6abad05507d315d1e5: stopping tablet replica
I20260929 19:03:03.921751 11134 tablet_replica.cc:330] T 18372107f12f454bad8ef2177b0269dd P 1094d0c4c0524fc7b6308499947dddb2: stopping tablet replica
I20260929 19:03:03.921747 10967 tablet_service.cc:1514] Processing DeleteTablet for tablet 18372107f12f454bad8ef2177b0269dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:03 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:36660
I20260929 19:03:03.921787 11135 raft_consensus.cc:2229] T 18372107f12f454bad8ef2177b0269dd P 5dce99a6cd474c6abad05507d315d1e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:03:03.921800 11134 raft_consensus.cc:2229] T 18372107f12f454bad8ef2177b0269dd P 1094d0c4c0524fc7b6308499947dddb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:03:03.921818 11136 tablet_replica.cc:330] T 18372107f12f454bad8ef2177b0269dd P 1b41ba80cad24d3db1d428f488d903d2: stopping tablet replica
I20260929 19:03:03.921823 11135 raft_consensus.cc:2258] T 18372107f12f454bad8ef2177b0269dd P 5dce99a6cd474c6abad05507d315d1e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:03:03.921837 11134 raft_consensus.cc:2258] T 18372107f12f454bad8ef2177b0269dd P 1094d0c4c0524fc7b6308499947dddb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:03:03.921865 11136 raft_consensus.cc:2229] T 18372107f12f454bad8ef2177b0269dd P 1b41ba80cad24d3db1d428f488d903d2 [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:03:03.921929 11136 raft_consensus.cc:2258] T 18372107f12f454bad8ef2177b0269dd P 1b41ba80cad24d3db1d428f488d903d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:03:03.921991 10550 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35888:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:03:03.922149 10550 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:03:03.922144 11134 ts_tablet_manager.cc:1905] T 18372107f12f454bad8ef2177b0269dd P 1094d0c4c0524fc7b6308499947dddb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:03:03.922602 11135 ts_tablet_manager.cc:1905] T 18372107f12f454bad8ef2177b0269dd P 5dce99a6cd474c6abad05507d315d1e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:03:03.923298 10687 tablet_service.cc:1467] Processing CreateTablet for tablet ca421ed3a7d64d92a4424a91b3725c29 (DEFAULT_TABLE table=test-table [id=290b7b139de64cd187b89886d3d62675]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:03.923295 10827 tablet_service.cc:1467] Processing CreateTablet for tablet ca421ed3a7d64d92a4424a91b3725c29 (DEFAULT_TABLE table=test-table [id=290b7b139de64cd187b89886d3d62675]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:03.923394 10827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca421ed3a7d64d92a4424a91b3725c29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:03.923463 10687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca421ed3a7d64d92a4424a91b3725c29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:03.923751 11134 ts_tablet_manager.cc:1918] T 18372107f12f454bad8ef2177b0269dd P 1094d0c4c0524fc7b6308499947dddb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 19:03:03.923794 11134 log.cc:1196] T 18372107f12f454bad8ef2177b0269dd P 1094d0c4c0524fc7b6308499947dddb2: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-1/wal/wals/18372107f12f454bad8ef2177b0269dd
I20260929 19:03:03.923993 11134 ts_tablet_manager.cc:1939] T 18372107f12f454bad8ef2177b0269dd P 1094d0c4c0524fc7b6308499947dddb2: Deleting consensus metadata
I20260929 19:03:03.924214 11135 ts_tablet_manager.cc:1918] T 18372107f12f454bad8ef2177b0269dd P 5dce99a6cd474c6abad05507d315d1e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 19:03:03.924259 11135 log.cc:1196] T 18372107f12f454bad8ef2177b0269dd P 5dce99a6cd474c6abad05507d315d1e5: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-0/wal/wals/18372107f12f454bad8ef2177b0269dd
I20260929 19:03:03.924458 11135 ts_tablet_manager.cc:1939] T 18372107f12f454bad8ef2177b0269dd P 5dce99a6cd474c6abad05507d315d1e5: Deleting consensus metadata
I20260929 19:03:03.924582 11145 tablet_bootstrap.cc:492] T ca421ed3a7d64d92a4424a91b3725c29 P 1094d0c4c0524fc7b6308499947dddb2: Bootstrap starting.
I20260929 19:03:03.924706 11136 ts_tablet_manager.cc:1905] T 18372107f12f454bad8ef2177b0269dd P 1b41ba80cad24d3db1d428f488d903d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:03:03.924867 10535 catalog_manager.cc:4841] TS 5dce99a6cd474c6abad05507d315d1e5 (127.8.158.65:33741): tablet 18372107f12f454bad8ef2177b0269dd (table test-table [id=0b5a972b6a134244937ab2e9b8d01f6f]) successfully deleted
I20260929 19:03:03.924992 11145 tablet_bootstrap.cc:654] T ca421ed3a7d64d92a4424a91b3725c29 P 1094d0c4c0524fc7b6308499947dddb2: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:03.925149 10536 catalog_manager.cc:4841] TS 1094d0c4c0524fc7b6308499947dddb2 (127.8.158.66:46157): tablet 18372107f12f454bad8ef2177b0269dd (table test-table [id=0b5a972b6a134244937ab2e9b8d01f6f]) successfully deleted
I20260929 19:03:03.925778 11145 tablet_bootstrap.cc:492] T ca421ed3a7d64d92a4424a91b3725c29 P 1094d0c4c0524fc7b6308499947dddb2: No bootstrap required, opened a new log
I20260929 19:03:03.925809 11145 ts_tablet_manager.cc:1397] T ca421ed3a7d64d92a4424a91b3725c29 P 1094d0c4c0524fc7b6308499947dddb2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:03.925904 11145 raft_consensus.cc:348] T ca421ed3a7d64d92a4424a91b3725c29 P 1094d0c4c0524fc7b6308499947dddb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } }
I20260929 19:03:03.925944 11145 raft_consensus.cc:374] T ca421ed3a7d64d92a4424a91b3725c29 P 1094d0c4c0524fc7b6308499947dddb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:03.925942 11136 ts_tablet_manager.cc:1918] T 18372107f12f454bad8ef2177b0269dd P 1b41ba80cad24d3db1d428f488d903d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 19:03:03.925958 11145 raft_consensus.cc:729] T ca421ed3a7d64d92a4424a91b3725c29 P 1094d0c4c0524fc7b6308499947dddb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1094d0c4c0524fc7b6308499947dddb2, State: Initialized, Role: FOLLOWER
I20260929 19:03:03.925978 11136 log.cc:1196] T 18372107f12f454bad8ef2177b0269dd P 1b41ba80cad24d3db1d428f488d903d2: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-2/wal/wals/18372107f12f454bad8ef2177b0269dd
I20260929 19:03:03.926020 11145 consensus_queue.cc:260] T ca421ed3a7d64d92a4424a91b3725c29 P 1094d0c4c0524fc7b6308499947dddb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } }
I20260929 19:03:03.926113 11145 ts_tablet_manager.cc:1428] T ca421ed3a7d64d92a4424a91b3725c29 P 1094d0c4c0524fc7b6308499947dddb2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:03.926153 11136 ts_tablet_manager.cc:1939] T 18372107f12f454bad8ef2177b0269dd P 1b41ba80cad24d3db1d428f488d903d2: Deleting consensus metadata
I20260929 19:03:03.926280 11146 tablet_bootstrap.cc:492] T ca421ed3a7d64d92a4424a91b3725c29 P 5dce99a6cd474c6abad05507d315d1e5: Bootstrap starting.
I20260929 19:03:03.926579 11146 tablet_bootstrap.cc:654] T ca421ed3a7d64d92a4424a91b3725c29 P 5dce99a6cd474c6abad05507d315d1e5: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:03.926589 10534 catalog_manager.cc:4841] TS 1b41ba80cad24d3db1d428f488d903d2 (127.8.158.67:38517): tablet 18372107f12f454bad8ef2177b0269dd (table test-table [id=0b5a972b6a134244937ab2e9b8d01f6f]) successfully deleted
I20260929 19:03:03.926844 10967 tablet_service.cc:1467] Processing CreateTablet for tablet ca421ed3a7d64d92a4424a91b3725c29 (DEFAULT_TABLE table=test-table [id=290b7b139de64cd187b89886d3d62675]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:03.926939 10967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca421ed3a7d64d92a4424a91b3725c29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:03.927978 11147 tablet_bootstrap.cc:492] T ca421ed3a7d64d92a4424a91b3725c29 P 1b41ba80cad24d3db1d428f488d903d2: Bootstrap starting.
I20260929 19:03:03.928102 11146 tablet_bootstrap.cc:492] T ca421ed3a7d64d92a4424a91b3725c29 P 5dce99a6cd474c6abad05507d315d1e5: No bootstrap required, opened a new log
I20260929 19:03:03.928154 11146 ts_tablet_manager.cc:1397] T ca421ed3a7d64d92a4424a91b3725c29 P 5dce99a6cd474c6abad05507d315d1e5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:03:03.928304 11147 tablet_bootstrap.cc:654] T ca421ed3a7d64d92a4424a91b3725c29 P 1b41ba80cad24d3db1d428f488d903d2: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:03.928278 11146 raft_consensus.cc:348] T ca421ed3a7d64d92a4424a91b3725c29 P 5dce99a6cd474c6abad05507d315d1e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } }
I20260929 19:03:03.928345 11146 raft_consensus.cc:374] T ca421ed3a7d64d92a4424a91b3725c29 P 5dce99a6cd474c6abad05507d315d1e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:03.928369 11146 raft_consensus.cc:729] T ca421ed3a7d64d92a4424a91b3725c29 P 5dce99a6cd474c6abad05507d315d1e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dce99a6cd474c6abad05507d315d1e5, State: Initialized, Role: FOLLOWER
I20260929 19:03:03.928419 11146 consensus_queue.cc:260] T ca421ed3a7d64d92a4424a91b3725c29 P 5dce99a6cd474c6abad05507d315d1e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } }
I20260929 19:03:03.928519 11146 ts_tablet_manager.cc:1428] T ca421ed3a7d64d92a4424a91b3725c29 P 5dce99a6cd474c6abad05507d315d1e5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:03.928930 11147 tablet_bootstrap.cc:492] T ca421ed3a7d64d92a4424a91b3725c29 P 1b41ba80cad24d3db1d428f488d903d2: No bootstrap required, opened a new log
I20260929 19:03:03.929000 11147 ts_tablet_manager.cc:1397] T ca421ed3a7d64d92a4424a91b3725c29 P 1b41ba80cad24d3db1d428f488d903d2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:03.929105 11147 raft_consensus.cc:348] T ca421ed3a7d64d92a4424a91b3725c29 P 1b41ba80cad24d3db1d428f488d903d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } }
I20260929 19:03:03.929189 11147 raft_consensus.cc:374] T ca421ed3a7d64d92a4424a91b3725c29 P 1b41ba80cad24d3db1d428f488d903d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:03.929214 11147 raft_consensus.cc:729] T ca421ed3a7d64d92a4424a91b3725c29 P 1b41ba80cad24d3db1d428f488d903d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b41ba80cad24d3db1d428f488d903d2, State: Initialized, Role: FOLLOWER
I20260929 19:03:03.929265 11147 consensus_queue.cc:260] T ca421ed3a7d64d92a4424a91b3725c29 P 1b41ba80cad24d3db1d428f488d903d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } }
I20260929 19:03:03.929358 11147 ts_tablet_manager.cc:1428] T ca421ed3a7d64d92a4424a91b3725c29 P 1b41ba80cad24d3db1d428f488d903d2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:04.264750 11117 raft_consensus.cc:482] T ca421ed3a7d64d92a4424a91b3725c29 P 5dce99a6cd474c6abad05507d315d1e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:03:04.264875 11117 raft_consensus.cc:504] T ca421ed3a7d64d92a4424a91b3725c29 P 5dce99a6cd474c6abad05507d315d1e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } }
I20260929 19:03:04.265090 11117 leader_election.cc:288] T ca421ed3a7d64d92a4424a91b3725c29 P 5dce99a6cd474c6abad05507d315d1e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1094d0c4c0524fc7b6308499947dddb2 (127.8.158.66:46157), 1b41ba80cad24d3db1d428f488d903d2 (127.8.158.67:38517)
I20260929 19:03:04.265339 10847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca421ed3a7d64d92a4424a91b3725c29" candidate_uuid: "5dce99a6cd474c6abad05507d315d1e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1094d0c4c0524fc7b6308499947dddb2" is_pre_election: true
I20260929 19:03:04.265419 10847 raft_consensus.cc:2454] T ca421ed3a7d64d92a4424a91b3725c29 P 1094d0c4c0524fc7b6308499947dddb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dce99a6cd474c6abad05507d315d1e5 in term 0.
I20260929 19:03:04.265347 10985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca421ed3a7d64d92a4424a91b3725c29" candidate_uuid: "5dce99a6cd474c6abad05507d315d1e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b41ba80cad24d3db1d428f488d903d2" is_pre_election: true
I20260929 19:03:04.265465 10985 raft_consensus.cc:2454] T ca421ed3a7d64d92a4424a91b3725c29 P 1b41ba80cad24d3db1d428f488d903d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dce99a6cd474c6abad05507d315d1e5 in term 0.
I20260929 19:03:04.265652 10638 leader_election.cc:302] T ca421ed3a7d64d92a4424a91b3725c29 P 5dce99a6cd474c6abad05507d315d1e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b41ba80cad24d3db1d428f488d903d2, 5dce99a6cd474c6abad05507d315d1e5; no voters:
I20260929 19:03:04.265759 11117 raft_consensus.cc:2781] T ca421ed3a7d64d92a4424a91b3725c29 P 5dce99a6cd474c6abad05507d315d1e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:03:04.265799 11117 raft_consensus.cc:482] T ca421ed3a7d64d92a4424a91b3725c29 P 5dce99a6cd474c6abad05507d315d1e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:03:04.265823 11117 raft_consensus.cc:3037] T ca421ed3a7d64d92a4424a91b3725c29 P 5dce99a6cd474c6abad05507d315d1e5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:04.266521 11117 raft_consensus.cc:504] T ca421ed3a7d64d92a4424a91b3725c29 P 5dce99a6cd474c6abad05507d315d1e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } }
I20260929 19:03:04.266657 11117 leader_election.cc:288] T ca421ed3a7d64d92a4424a91b3725c29 P 5dce99a6cd474c6abad05507d315d1e5 [CANDIDATE]: Term 1 election: Requested vote from peers 1094d0c4c0524fc7b6308499947dddb2 (127.8.158.66:46157), 1b41ba80cad24d3db1d428f488d903d2 (127.8.158.67:38517)
I20260929 19:03:04.266785 10847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca421ed3a7d64d92a4424a91b3725c29" candidate_uuid: "5dce99a6cd474c6abad05507d315d1e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1094d0c4c0524fc7b6308499947dddb2"
I20260929 19:03:04.266862 10847 raft_consensus.cc:3037] T ca421ed3a7d64d92a4424a91b3725c29 P 1094d0c4c0524fc7b6308499947dddb2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:04.266871 10985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca421ed3a7d64d92a4424a91b3725c29" candidate_uuid: "5dce99a6cd474c6abad05507d315d1e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b41ba80cad24d3db1d428f488d903d2"
I20260929 19:03:04.266940 10985 raft_consensus.cc:3037] T ca421ed3a7d64d92a4424a91b3725c29 P 1b41ba80cad24d3db1d428f488d903d2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:04.267608 10847 raft_consensus.cc:2454] T ca421ed3a7d64d92a4424a91b3725c29 P 1094d0c4c0524fc7b6308499947dddb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dce99a6cd474c6abad05507d315d1e5 in term 1.
I20260929 19:03:04.267704 10985 raft_consensus.cc:2454] T ca421ed3a7d64d92a4424a91b3725c29 P 1b41ba80cad24d3db1d428f488d903d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dce99a6cd474c6abad05507d315d1e5 in term 1.
I20260929 19:03:04.267865 10640 leader_election.cc:302] T ca421ed3a7d64d92a4424a91b3725c29 P 5dce99a6cd474c6abad05507d315d1e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1094d0c4c0524fc7b6308499947dddb2, 5dce99a6cd474c6abad05507d315d1e5; no voters:
I20260929 19:03:04.267956 11117 raft_consensus.cc:2781] T ca421ed3a7d64d92a4424a91b3725c29 P 5dce99a6cd474c6abad05507d315d1e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:03:04.268033 11117 raft_consensus.cc:686] T ca421ed3a7d64d92a4424a91b3725c29 P 5dce99a6cd474c6abad05507d315d1e5 [term 1 LEADER]: Becoming Leader. State: Replica: 5dce99a6cd474c6abad05507d315d1e5, State: Running, Role: LEADER
I20260929 19:03:04.268090 11117 consensus_queue.cc:237] T ca421ed3a7d64d92a4424a91b3725c29 P 5dce99a6cd474c6abad05507d315d1e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } }
I20260929 19:03:04.268698 10547 catalog_manager.cc:5495] T ca421ed3a7d64d92a4424a91b3725c29 P 5dce99a6cd474c6abad05507d315d1e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dce99a6cd474c6abad05507d315d1e5 (127.8.158.65). New cstate: current_term: 1 leader_uuid: "5dce99a6cd474c6abad05507d315d1e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dce99a6cd474c6abad05507d315d1e5" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 33741 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 } health_report { overall_health: UNKNOWN } } }
I20260929 19:03:04.282686 11165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:04.280217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49878 (local address 127.8.158.65:33741)
0929 19:03:04.280385 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:04.280387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:04.280394 (+ 7us) server_negotiation.cc:407] Connection header received
0929 19:03:04.280416 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:04.280416 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:04.280435 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:04.280513 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:03:04.280669 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:04.281276 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:04.282000 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:04.282198 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:04.282253 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:04.282428 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:03:04.282453 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:03:04.282499 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:04.282533 (+ 34us) server_negotiation.cc:299] Negotiation successful
0929 19:03:04.282548 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":98,"threads_started":1}
I20260929 19:03:04.284910 10847 raft_consensus.cc:1261] T ca421ed3a7d64d92a4424a91b3725c29 P 1094d0c4c0524fc7b6308499947dddb2 [term 1 FOLLOWER]: Refusing update from remote peer 5dce99a6cd474c6abad05507d315d1e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:03:04.284932 10985 raft_consensus.cc:1261] T ca421ed3a7d64d92a4424a91b3725c29 P 1b41ba80cad24d3db1d428f488d903d2 [term 1 FOLLOWER]: Refusing update from remote peer 5dce99a6cd474c6abad05507d315d1e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:03:04.285138 11097 consensus_queue.cc:1035] T ca421ed3a7d64d92a4424a91b3725c29 P 5dce99a6cd474c6abad05507d315d1e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1094d0c4c0524fc7b6308499947dddb2" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:03:04.285234 11167 consensus_queue.cc:1035] T ca421ed3a7d64d92a4424a91b3725c29 P 5dce99a6cd474c6abad05507d315d1e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b41ba80cad24d3db1d428f488d903d2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 19:03:04.289376 11115 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 19:03:05.808296 10547 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:03:05.808410 10547 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:03:05.809361 10547 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 06a804e1e1b34f37b46d8763e45660ad: Sending DeleteTablet for 3 replicas of tablet ca421ed3a7d64d92a4424a91b3725c29
I20260929 19:03:05.809669 10686 tablet_service.cc:1514] Processing DeleteTablet for tablet ca421ed3a7d64d92a4424a91b3725c29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:05 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:49858
I20260929 19:03:05.809667 10827 tablet_service.cc:1514] Processing DeleteTablet for tablet ca421ed3a7d64d92a4424a91b3725c29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:05 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:45686
I20260929 19:03:05.809881 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 10622
I20260929 19:03:05.810359 10967 tablet_service.cc:1514] Processing DeleteTablet for tablet ca421ed3a7d64d92a4424a91b3725c29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:05 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:36660
I20260929 19:03:05.810624 11189 tablet_replica.cc:330] T ca421ed3a7d64d92a4424a91b3725c29 P 1094d0c4c0524fc7b6308499947dddb2: stopping tablet replica
I20260929 19:03:05.810690 11189 raft_consensus.cc:2229] T ca421ed3a7d64d92a4424a91b3725c29 P 1094d0c4c0524fc7b6308499947dddb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:03:05.810732 11189 raft_consensus.cc:2258] T ca421ed3a7d64d92a4424a91b3725c29 P 1094d0c4c0524fc7b6308499947dddb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:03:05.810854 11191 tablet_replica.cc:330] T ca421ed3a7d64d92a4424a91b3725c29 P 1b41ba80cad24d3db1d428f488d903d2: stopping tablet replica
I20260929 19:03:05.810911 11191 raft_consensus.cc:2229] T ca421ed3a7d64d92a4424a91b3725c29 P 1b41ba80cad24d3db1d428f488d903d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:03:05.810952 11191 raft_consensus.cc:2258] T ca421ed3a7d64d92a4424a91b3725c29 P 1b41ba80cad24d3db1d428f488d903d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:03:05.811157 11189 ts_tablet_manager.cc:1905] T ca421ed3a7d64d92a4424a91b3725c29 P 1094d0c4c0524fc7b6308499947dddb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:03:05.811411 11191 ts_tablet_manager.cc:1905] T ca421ed3a7d64d92a4424a91b3725c29 P 1b41ba80cad24d3db1d428f488d903d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:03:05.812912 11189 ts_tablet_manager.cc:1918] T ca421ed3a7d64d92a4424a91b3725c29 P 1094d0c4c0524fc7b6308499947dddb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 19:03:05.812996 11189 log.cc:1196] T ca421ed3a7d64d92a4424a91b3725c29 P 1094d0c4c0524fc7b6308499947dddb2: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-1/wal/wals/ca421ed3a7d64d92a4424a91b3725c29
I20260929 19:03:05.813191 11191 ts_tablet_manager.cc:1918] T ca421ed3a7d64d92a4424a91b3725c29 P 1b41ba80cad24d3db1d428f488d903d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 19:03:05.813261 11191 log.cc:1196] T ca421ed3a7d64d92a4424a91b3725c29 P 1b41ba80cad24d3db1d428f488d903d2: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708573736807-8825-0/minicluster-data/ts-2/wal/wals/ca421ed3a7d64d92a4424a91b3725c29
I20260929 19:03:05.813380 11189 ts_tablet_manager.cc:1939] T ca421ed3a7d64d92a4424a91b3725c29 P 1094d0c4c0524fc7b6308499947dddb2: Deleting consensus metadata
I20260929 19:03:05.813520 11191 ts_tablet_manager.cc:1939] T ca421ed3a7d64d92a4424a91b3725c29 P 1b41ba80cad24d3db1d428f488d903d2: Deleting consensus metadata
I20260929 19:03:05.813724 10536 catalog_manager.cc:4841] TS 1094d0c4c0524fc7b6308499947dddb2 (127.8.158.66:46157): tablet ca421ed3a7d64d92a4424a91b3725c29 (table test-table [id=290b7b139de64cd187b89886d3d62675]) successfully deleted
I20260929 19:03:05.813834 10534 catalog_manager.cc:4841] TS 1b41ba80cad24d3db1d428f488d903d2 (127.8.158.67:38517): tablet ca421ed3a7d64d92a4424a91b3725c29 (table test-table [id=290b7b139de64cd187b89886d3d62675]) successfully deleted
W20260929 19:03:05.818050 10535 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.158.65:33741 (error 108)
I20260929 19:03:05.818279 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 10762
I20260929 19:03:05.819195 11193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:05.818972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:33741 (local address 127.0.0.1:49888)
0929 19:03:05.819116 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:05.819162 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.65:33741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":71,"threads_started":1}
W20260929 19:03:05.822474 10535 catalog_manager.cc:4584] TS 5dce99a6cd474c6abad05507d315d1e5 (127.8.158.65:33741): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ca421ed3a7d64d92a4424a91b3725c29: Network error: Client connection negotiation failed: client connection to 127.8.158.65:33741: connect: Connection refused (error 111)
I20260929 19:03:05.823143 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 10902
I20260929 19:03:05.828810 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 10518
2026-09-29T19:03:05Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6048 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:03:05 dist-test-slave-nmsz krb5kdc[11196](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:03:05 dist-test-slave-nmsz krb5kdc[11196](info): set up 2 sockets
Sep 29 19:03:05 dist-test-slave-nmsz krb5kdc[11196](info): commencing operation
krb5kdc: starting...
W20260929 19:03:06.890019 8825 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Sep 29 19:03:06 dist-test-slave-nmsz krb5kdc[11196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:03:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:03:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.158.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
I20260929 19:03:07.001931 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:36715
--webserver_interface=127.8.158.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:40781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:36715
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:07.080541 11209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:07.080727 11209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:07.080749 11209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:07.081797 11209 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:03:07.081856 11209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:07.081871 11209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:07.081884 11209 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:03:07.081897 11209 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:03:07.083268 11209 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:40781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:36715
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:36715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.158.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:07.083451 11209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:07.083640 11209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:07.088135 11217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:07.088181 11214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:07.088186 11215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:07.088364 11209 server_base.cc:1034] running on GCE node
I20260929 19:03:07.088601 11209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:07.088872 11209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:07.090036 11209 hybrid_clock.cc:648] HybridClock initialized: now 1790708587090028 us; error 43 us; skew 500 ppm
Sep 29 19:03:07 dist-test-slave-nmsz krb5kdc[11196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708587, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:07.092593 11209 init.cc:377] Logged in from keytab as kudu/127.8.158.126@KRBTEST.COM (short username kudu)
I20260929 19:03:07.092990 11209 webserver.cc:460] Webserver started at http://127.8.158.126:33189/ using document root <none> and password file <none>
I20260929 19:03:07.093209 11209 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:07.093274 11209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:07.093432 11209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:07.094496 11209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "dc6eb938b4254f519e8dcc7c6cb1f4af"
format_stamp: "Formatted at 2026-09-29 19:03:07 on dist-test-slave-nmsz"
server_key: "d30f67b7f6a595958c337e429c0c3a1c"
server_key_iv: "d474cdb3702894ebadbfe5c7634a4547"
server_key_version: "encryptionkey@0"
I20260929 19:03:07.094841 11209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "dc6eb938b4254f519e8dcc7c6cb1f4af"
format_stamp: "Formatted at 2026-09-29 19:03:07 on dist-test-slave-nmsz"
server_key: "d30f67b7f6a595958c337e429c0c3a1c"
server_key_iv: "d474cdb3702894ebadbfe5c7634a4547"
server_key_version: "encryptionkey@0"
I20260929 19:03:07.096119 11209 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:07.096823 11224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:07.096997 11209 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:03:07.097091 11209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "dc6eb938b4254f519e8dcc7c6cb1f4af"
format_stamp: "Formatted at 2026-09-29 19:03:07 on dist-test-slave-nmsz"
server_key: "d30f67b7f6a595958c337e429c0c3a1c"
server_key_iv: "d474cdb3702894ebadbfe5c7634a4547"
server_key_version: "encryptionkey@0"
I20260929 19:03:07.097170 11209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:07.132929 11209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:07.134828 11209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:07.134989 11209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:07.138976 11209 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.126:36715
I20260929 19:03:07.139007 11276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.126:36715 every 8 connection(s)
I20260929 19:03:07.139436 11209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:03:07.140488 11277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:07.142645 11277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc6eb938b4254f519e8dcc7c6cb1f4af: Bootstrap starting.
I20260929 19:03:07.143205 11277 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc6eb938b4254f519e8dcc7c6cb1f4af: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:07.143496 11277 log.cc:824] T 00000000000000000000000000000000 P dc6eb938b4254f519e8dcc7c6cb1f4af: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:07.144178 11277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc6eb938b4254f519e8dcc7c6cb1f4af: No bootstrap required, opened a new log
I20260929 19:03:07.145704 11277 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc6eb938b4254f519e8dcc7c6cb1f4af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6eb938b4254f519e8dcc7c6cb1f4af" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 36715 } }
I20260929 19:03:07.145860 11277 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc6eb938b4254f519e8dcc7c6cb1f4af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:07.145908 11277 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc6eb938b4254f519e8dcc7c6cb1f4af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc6eb938b4254f519e8dcc7c6cb1f4af, State: Initialized, Role: FOLLOWER
I20260929 19:03:07.146051 11277 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc6eb938b4254f519e8dcc7c6cb1f4af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6eb938b4254f519e8dcc7c6cb1f4af" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 36715 } }
I20260929 19:03:07.146132 11277 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc6eb938b4254f519e8dcc7c6cb1f4af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:03:07.146176 11277 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc6eb938b4254f519e8dcc7c6cb1f4af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:03:07.146245 11277 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc6eb938b4254f519e8dcc7c6cb1f4af [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:07.146445 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 11209
I20260929 19:03:07.146528 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:03:07.146683 8825 external_mini_cluster.cc:1428] Setting key f9254d9ddc8fbfbfa6195468b6261036
I20260929 19:03:07.147006 11277 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc6eb938b4254f519e8dcc7c6cb1f4af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6eb938b4254f519e8dcc7c6cb1f4af" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 36715 } }
I20260929 19:03:07.147109 11277 leader_election.cc:302] T 00000000000000000000000000000000 P dc6eb938b4254f519e8dcc7c6cb1f4af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc6eb938b4254f519e8dcc7c6cb1f4af; no voters:
I20260929 19:03:07.147269 11277 leader_election.cc:288] T 00000000000000000000000000000000 P dc6eb938b4254f519e8dcc7c6cb1f4af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:03:07.147307 11282 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc6eb938b4254f519e8dcc7c6cb1f4af [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:03:07.147447 11282 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc6eb938b4254f519e8dcc7c6cb1f4af [term 1 LEADER]: Becoming Leader. State: Replica: dc6eb938b4254f519e8dcc7c6cb1f4af, State: Running, Role: LEADER
I20260929 19:03:07.147531 11277 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc6eb938b4254f519e8dcc7c6cb1f4af [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:03:07.147540 11282 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc6eb938b4254f519e8dcc7c6cb1f4af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6eb938b4254f519e8dcc7c6cb1f4af" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 36715 } }
I20260929 19:03:07.148300 11282 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc6eb938b4254f519e8dcc7c6cb1f4af [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc6eb938b4254f519e8dcc7c6cb1f4af. Latest consensus state: current_term: 1 leader_uuid: "dc6eb938b4254f519e8dcc7c6cb1f4af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6eb938b4254f519e8dcc7c6cb1f4af" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 36715 } } }
I20260929 19:03:07.148383 11282 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc6eb938b4254f519e8dcc7c6cb1f4af [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:07.148587 11291 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:03:07.148713 11283 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc6eb938b4254f519e8dcc7c6cb1f4af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc6eb938b4254f519e8dcc7c6cb1f4af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6eb938b4254f519e8dcc7c6cb1f4af" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 36715 } } }
I20260929 19:03:07.148802 11283 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc6eb938b4254f519e8dcc7c6cb1f4af [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:07.149466 11291 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:03:07.151003 11291 catalog_manager.cc:1347] Generated new cluster ID: e730dfa821f64b18a8dffa5fb4071a0d
I20260929 19:03:07.151060 11291 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 19:03:07 dist-test-slave-nmsz krb5kdc[11196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:03:07.156900 11291 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:03:07.157549 11281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:07.147137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45374 (local address 127.8.158.126:36715)
0929 19:03:07.147323 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:07.147332 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:07.147367 (+ 35us) server_negotiation.cc:407] Connection header received
0929 19:03:07.147566 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:07.147574 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:07.147628 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:07.148590 (+ 962us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:07.148712 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:07.150240 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:07.151284 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:07.151695 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:07.151825 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:07.154055 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:07.154059 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:07.154059 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:07.154070 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:07.155758 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:07.156124 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:07.156127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:07.156128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:07.156232 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:07.156464 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:07.156465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:07.156465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:07.156783 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:07.156961 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:07.157222 (+ 261us) server_negotiation.cc:299] Negotiation successful
0929 19:03:07.157365 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":73,"threads_started":1}
I20260929 19:03:07.157562 11291 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:03:07.160090 11291 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc6eb938b4254f519e8dcc7c6cb1f4af: Generated new TSK 0
I20260929 19:03:07.160238 11291 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.158.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab.
Entry for principal kudu/127.8.158.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab.
I20260929 19:03:07.187321 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.65:0
--local_ip_for_outbound_sockets=127.8.158.65
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:36715
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab
--principal=kudu/127.8.158.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:40781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:07.283314 11303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:07.283504 11303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:07.283525 11303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:07.284667 11303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:07.284734 11303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:07.284780 11303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.65
I20260929 19:03:07.286165 11303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:40781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab
--principal=kudu/127.8.158.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:36715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.65
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:07.286449 11303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:07.286667 11303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:07.291297 11309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:07.291316 11311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:07.291447 11303 server_base.cc:1034] running on GCE node
W20260929 19:03:07.291316 11308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:07.291702 11303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:07.291939 11303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:07.293099 11303 hybrid_clock.cc:648] HybridClock initialized: now 1790708587293087 us; error 31 us; skew 500 ppm
Sep 29 19:03:07 dist-test-slave-nmsz krb5kdc[11196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708587, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:07.295473 11303 init.cc:377] Logged in from keytab as kudu/127.8.158.65@KRBTEST.COM (short username kudu)
I20260929 19:03:07.295928 11303 webserver.cc:460] Webserver started at http://127.8.158.65:36431/ using document root <none> and password file <none>
I20260929 19:03:07.296162 11303 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:07.296207 11303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:07.296325 11303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:07.297314 11303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-0/data/instance:
uuid: "d9c2be5dc0f9443e83a30cd45473ab58"
format_stamp: "Formatted at 2026-09-29 19:03:07 on dist-test-slave-nmsz"
server_key: "f6263b07a2117ac0cbacc368f421d793"
server_key_iv: "31272ec439743dbc89c90829bafe28ac"
server_key_version: "encryptionkey@0"
I20260929 19:03:07.297653 11303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance:
uuid: "d9c2be5dc0f9443e83a30cd45473ab58"
format_stamp: "Formatted at 2026-09-29 19:03:07 on dist-test-slave-nmsz"
server_key: "f6263b07a2117ac0cbacc368f421d793"
server_key_iv: "31272ec439743dbc89c90829bafe28ac"
server_key_version: "encryptionkey@0"
I20260929 19:03:07.299001 11303 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:03:07.299799 11318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:07.299954 11303 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:03:07.300055 11303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-0/wal
uuid: "d9c2be5dc0f9443e83a30cd45473ab58"
format_stamp: "Formatted at 2026-09-29 19:03:07 on dist-test-slave-nmsz"
server_key: "f6263b07a2117ac0cbacc368f421d793"
server_key_iv: "31272ec439743dbc89c90829bafe28ac"
server_key_version: "encryptionkey@0"
I20260929 19:03:07.300122 11303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:07.332305 11303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:07.334656 11303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:07.334836 11303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:07.335072 11303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:07.335376 11303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:07.335410 11303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:07.335433 11303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:07.335446 11303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:07.341012 11303 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.65:40951
I20260929 19:03:07.341063 11431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.65:40951 every 8 connection(s)
I20260929 19:03:07.341354 11303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
I20260929 19:03:07.342303 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 11303
I20260929 19:03:07.342453 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance
I20260929 19:03:07.342623 8825 external_mini_cluster.cc:1428] Setting key dc0c112d883b50eae186e942de0bfdb9
Sep 29 19:03:07 dist-test-slave-nmsz krb5kdc[11196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708587, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.158.66@KRBTEST.COM; defaulting to no policy
I20260929 19:03:07.351202 11281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:07.342003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:55323 (local address 127.8.158.126:36715)
0929 19:03:07.342074 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:07.342076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:07.342274 (+ 198us) server_negotiation.cc:407] Connection header received
0929 19:03:07.342884 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:07.342885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:07.342898 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:07.342979 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:07.343978 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:07.344698 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:07.345693 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:07.345906 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:07.345991 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:07.348300 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:07.348301 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:07.348301 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:07.348315 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:07.349709 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:07.350129 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:07.350131 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:07.350131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:07.350190 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:07.350563 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:07.350564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:07.350564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:07.350670 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:07.350747 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:07.351018 (+ 271us) server_negotiation.cc:299] Negotiation successful
0929 19:03:07.351098 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 19:03:07.351392 11434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:07.341999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:36715 (local address 127.8.158.65:55323)
0929 19:03:07.342135 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:07.342174 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:07.342730 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:07.343065 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:07.343069 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:07.343689 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:07.343871 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:07.343876 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:07.344812 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:07.344814 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:07.345595 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:07.345597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:07.346002 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:07.346003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:07.346135 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:07.346635 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:07.346643 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:07.348180 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:07.349812 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:07.349819 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:07.349821 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:07.350037 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:07.350256 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:07.350257 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:07.350258 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:07.350441 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:07.350750 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:07.350756 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:07.350887 (+ 131us) client_negotiation.cc:769] Sending connection context
0929 19:03:07.350983 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 19:03:07.351126 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":73,"threads_started":1}
I20260929 19:03:07.351828 11432 heartbeater.cc:347] Connected to a master server at 127.8.158.126:36715
I20260929 19:03:07.351900 11432 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:07.352089 11432 heartbeater.cc:511] Master 127.8.158.126:36715 requested a full tablet report, sending...
I20260929 19:03:07.352568 11241 ts_manager.cc:194] Registered new tserver with Master: d9c2be5dc0f9443e83a30cd45473ab58 (127.8.158.65:40951)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.66@KRBTEST.COM" created.
I20260929 19:03:07.353394 11241 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.65@KRBTEST.COM'} at 127.8.158.65:55323
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab.
Entry for principal kudu/127.8.158.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab.
I20260929 19:03:07.366568 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.66:0
--local_ip_for_outbound_sockets=127.8.158.66
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:36715
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab
--principal=kudu/127.8.158.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:40781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:07.444060 11437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:07.444250 11437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:07.444274 11437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:07.445341 11437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:07.445375 11437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:07.445407 11437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.66
I20260929 19:03:07.446740 11437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:40781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab
--principal=kudu/127.8.158.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:36715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.66
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:07.446916 11437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:07.447072 11437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:07.451874 11443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:07.451977 11445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:07.451977 11442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:07.452340 11437 server_base.cc:1034] running on GCE node
I20260929 19:03:07.452546 11437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:07.452745 11437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:07.453881 11437 hybrid_clock.cc:648] HybridClock initialized: now 1790708587453857 us; error 38 us; skew 500 ppm
Sep 29 19:03:07 dist-test-slave-nmsz krb5kdc[11196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708587, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:07.456274 11437 init.cc:377] Logged in from keytab as kudu/127.8.158.66@KRBTEST.COM (short username kudu)
I20260929 19:03:07.456719 11437 webserver.cc:460] Webserver started at http://127.8.158.66:33485/ using document root <none> and password file <none>
I20260929 19:03:07.456948 11437 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:07.457015 11437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:07.457161 11437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:07.458148 11437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-1/data/instance:
uuid: "4f55738cfd23429e86fde03a45b56ff6"
format_stamp: "Formatted at 2026-09-29 19:03:07 on dist-test-slave-nmsz"
server_key: "743538bb5a8570dc6a949c0dcce93903"
server_key_iv: "3509b212810005404e75702d1d4a8cba"
server_key_version: "encryptionkey@0"
I20260929 19:03:07.458523 11437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance:
uuid: "4f55738cfd23429e86fde03a45b56ff6"
format_stamp: "Formatted at 2026-09-29 19:03:07 on dist-test-slave-nmsz"
server_key: "743538bb5a8570dc6a949c0dcce93903"
server_key_iv: "3509b212810005404e75702d1d4a8cba"
server_key_version: "encryptionkey@0"
I20260929 19:03:07.459800 11437 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:03:07.460506 11452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:07.460671 11437 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:03:07.460763 11437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-1/wal
uuid: "4f55738cfd23429e86fde03a45b56ff6"
format_stamp: "Formatted at 2026-09-29 19:03:07 on dist-test-slave-nmsz"
server_key: "743538bb5a8570dc6a949c0dcce93903"
server_key_iv: "3509b212810005404e75702d1d4a8cba"
server_key_version: "encryptionkey@0"
I20260929 19:03:07.460850 11437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:07.472304 11437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:07.474594 11437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:07.474753 11437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:07.474975 11437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:07.475265 11437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:07.475314 11437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:07.475347 11437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:07.475376 11437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:07.481194 11437 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.66:41329
I20260929 19:03:07.481242 11565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.66:41329 every 8 connection(s)
I20260929 19:03:07.481529 11437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
Sep 29 19:03:07 dist-test-slave-nmsz krb5kdc[11196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708587, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.66@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:03:07.490038 11281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:07.482185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:54125 (local address 127.8.158.126:36715)
0929 19:03:07.482261 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:07.482263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:07.482465 (+ 202us) server_negotiation.cc:407] Connection header received
0929 19:03:07.482918 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:07.482919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:07.482928 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:07.482979 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:07.483828 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:07.484299 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:07.485362 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:07.485535 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:07.485588 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:07.487570 (+ 1982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:07.487571 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:07.487571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:07.487577 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:07.488911 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:07.489294 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:07.489295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:07.489295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:07.489348 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:07.489568 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:07.489568 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:07.489569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:07.489661 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:07.489709 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:07.489921 (+ 212us) server_negotiation.cc:299] Negotiation successful
0929 19:03:07.489978 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 19:03:07.490293 11568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:07.482144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:36715 (local address 127.8.158.66:54125)
0929 19:03:07.482302 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:07.482374 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:07.482837 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:07.483029 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:07.483032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:07.483568 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:07.483754 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:07.483758 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:07.484379 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:07.484380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:07.485280 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:07.485283 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:07.485609 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:07.485610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:07.485722 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:07.486108 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:07.486116 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:07.487482 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:07.488994 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:07.488999 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:07.489001 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:07.489223 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:07.489406 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:07.489406 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:07.489407 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:07.489508 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:07.489714 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:07.489718 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:07.489822 (+ 104us) client_negotiation.cc:769] Sending connection context
0929 19:03:07.489898 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 19:03:07.490005 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":92,"threads_started":1}
I20260929 19:03:07.490595 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 11437
I20260929 19:03:07.490690 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance
I20260929 19:03:07.490788 11566 heartbeater.cc:347] Connected to a master server at 127.8.158.126:36715
I20260929 19:03:07.490902 11566 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:07.490902 8825 external_mini_cluster.cc:1428] Setting key 5e1f129170af5af640beb627e6c31329
I20260929 19:03:07.491067 11566 heartbeater.cc:511] Master 127.8.158.126:36715 requested a full tablet report, sending...
I20260929 19:03:07.491493 11240 ts_manager.cc:194] Registered new tserver with Master: 4f55738cfd23429e86fde03a45b56ff6 (127.8.158.66:41329)
I20260929 19:03:07.491966 11240 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.66@KRBTEST.COM'} at 127.8.158.66:54125
WARNING: no policy specified for kudu/127.8.158.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab.
Entry for principal kudu/127.8.158.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab.
I20260929 19:03:07.515458 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.67:0
--local_ip_for_outbound_sockets=127.8.158.67
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:36715
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab
--principal=kudu/127.8.158.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:40781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:07.594453 11571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:07.594637 11571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:07.594659 11571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:07.595801 11571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:07.595844 11571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:07.595870 11571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.67
I20260929 19:03:07.597239 11571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:40781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab
--principal=kudu/127.8.158.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:36715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.67
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:07.597467 11571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:07.597675 11571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:07.602420 11576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:07.602617 11571 server_base.cc:1034] running on GCE node
W20260929 19:03:07.602454 11579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:07.602411 11577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:07.602911 11571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:07.603165 11571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:07.604308 11571 hybrid_clock.cc:648] HybridClock initialized: now 1790708587604303 us; error 41 us; skew 500 ppm
Sep 29 19:03:07 dist-test-slave-nmsz krb5kdc[11196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708587, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:07.606829 11571 init.cc:377] Logged in from keytab as kudu/127.8.158.67@KRBTEST.COM (short username kudu)
I20260929 19:03:07.607246 11571 webserver.cc:460] Webserver started at http://127.8.158.67:41437/ using document root <none> and password file <none>
I20260929 19:03:07.607422 11571 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:07.607462 11571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:07.607547 11571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:07.608465 11571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-2/data/instance:
uuid: "a077da573adb4a8d95ec6cd8227c420c"
format_stamp: "Formatted at 2026-09-29 19:03:07 on dist-test-slave-nmsz"
server_key: "71217f7e4b6758cb74496190ee9db6c6"
server_key_iv: "d69b99e1e136e5bf438ba5998544bcd8"
server_key_version: "encryptionkey@0"
I20260929 19:03:07.608752 11571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance:
uuid: "a077da573adb4a8d95ec6cd8227c420c"
format_stamp: "Formatted at 2026-09-29 19:03:07 on dist-test-slave-nmsz"
server_key: "71217f7e4b6758cb74496190ee9db6c6"
server_key_iv: "d69b99e1e136e5bf438ba5998544bcd8"
server_key_version: "encryptionkey@0"
I20260929 19:03:07.610049 11571 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:03:07.610770 11586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:07.610940 11571 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:07.611035 11571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-2/wal
uuid: "a077da573adb4a8d95ec6cd8227c420c"
format_stamp: "Formatted at 2026-09-29 19:03:07 on dist-test-slave-nmsz"
server_key: "71217f7e4b6758cb74496190ee9db6c6"
server_key_iv: "d69b99e1e136e5bf438ba5998544bcd8"
server_key_version: "encryptionkey@0"
I20260929 19:03:07.611114 11571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:07.630702 11571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:07.633018 11571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:07.633181 11571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:07.633407 11571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:07.633715 11571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:07.633770 11571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:07.633805 11571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:07.633833 11571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:07.639433 11571 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.67:43099
I20260929 19:03:07.639496 11699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.67:43099 every 8 connection(s)
I20260929 19:03:07.639812 11571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
Sep 29 19:03:07 dist-test-slave-nmsz krb5kdc[11196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708587, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.67@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:03:07.648650 11281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:07.640505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:55129 (local address 127.8.158.126:36715)
0929 19:03:07.640562 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:07.640578 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:07.640752 (+ 174us) server_negotiation.cc:407] Connection header received
0929 19:03:07.641256 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:07.641257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:07.641269 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:07.641360 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:07.642245 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:07.642789 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:07.643747 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:07.643947 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:07.644024 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:07.645928 (+ 1904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:07.645929 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:07.645929 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:07.645936 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:07.647159 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:07.647553 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:07.647554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:07.647555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:07.647610 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:07.647857 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:07.647858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:07.647859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:07.647990 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:07.648064 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:07.648496 (+ 432us) server_negotiation.cc:299] Negotiation successful
0929 19:03:07.648568 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 19:03:07.648701 11702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:07.640471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:36715 (local address 127.8.158.67:55129)
0929 19:03:07.640641 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:07.640673 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:07.641154 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:07.641397 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:07.641401 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:07.641968 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:07.642145 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:07.642150 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:07.642880 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:07.642882 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:07.643644 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:07.643646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:07.644039 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:07.644040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:07.644150 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:07.644569 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:07.644577 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:07.645824 (+ 1247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:07.647243 (+ 1419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:07.647249 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:07.647251 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:07.647464 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:07.647675 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:07.647676 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:07.647677 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:07.647775 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:07.648055 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:07.648061 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:07.648190 (+ 129us) client_negotiation.cc:769] Sending connection context
0929 19:03:07.648295 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 19:03:07.648444 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":88,"threads_started":1}
I20260929 19:03:07.649279 11700 heartbeater.cc:347] Connected to a master server at 127.8.158.126:36715
I20260929 19:03:07.649385 11700 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:07.649552 11700 heartbeater.cc:511] Master 127.8.158.126:36715 requested a full tablet report, sending...
I20260929 19:03:07.649771 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 11571
I20260929 19:03:07.649863 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance
I20260929 19:03:07.649914 11240 ts_manager.cc:194] Registered new tserver with Master: a077da573adb4a8d95ec6cd8227c420c (127.8.158.67:43099)
I20260929 19:03:07.650063 8825 external_mini_cluster.cc:1428] Setting key 5b0b5554614d72e15e634bbac4b79cec
I20260929 19:03:07.650323 11240 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.67@KRBTEST.COM'} at 127.8.158.67:55129
I20260929 19:03:07.651077 8825 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 19:03:07.661700 11281 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:07.660794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45380 (local address 127.8.158.126:36715)
0929 19:03:07.660865 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:07.660868 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:07.660986 (+ 118us) server_negotiation.cc:407] Connection header received
0929 19:03:07.661064 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:07.661065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:07.661080 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:07.661143 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:07.661634 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45380: BlockingRecv error: recv got EOF from 127.0.0.1:45380 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 19:03:07.662595 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 11303
I20260929 19:03:07.667109 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 11437
I20260929 19:03:07.671320 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 11571
I20260929 19:03:07.675494 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 11209
2026-09-29T19:03:07Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1845 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708573736807-8825-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:03:07 dist-test-slave-nmsz krb5kdc[11712](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:03:07 dist-test-slave-nmsz krb5kdc[11712](info): set up 2 sockets
Sep 29 19:03:07 dist-test-slave-nmsz krb5kdc[11712](info): commencing operation
krb5kdc: starting...
W20260929 19:03:08.733999 8825 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Sep 29 19:03:08 dist-test-slave-nmsz krb5kdc[11712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:03:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:03:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.158.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
I20260929 19:03:08.845923 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:34935
--webserver_interface=127.8.158.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:40767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:34935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:08.924358 11725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:08.924556 11725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:08.924577 11725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:08.925634 11725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:03:08.925669 11725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:08.925683 11725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:08.925696 11725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:03:08.925709 11725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:03:08.927101 11725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:40767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:34935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:34935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.158.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:08.927279 11725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:08.927428 11725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:08.932148 11731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:08.932197 11730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:08.932148 11733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:08.932365 11725 server_base.cc:1034] running on GCE node
I20260929 19:03:08.932641 11725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:08.932932 11725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:08.934125 11725 hybrid_clock.cc:648] HybridClock initialized: now 1790708588934113 us; error 31 us; skew 500 ppm
Sep 29 19:03:08 dist-test-slave-nmsz krb5kdc[11712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708588, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:08.941679 11725 init.cc:377] Logged in from keytab as kudu/127.8.158.126@KRBTEST.COM (short username kudu)
I20260929 19:03:08.942421 11725 webserver.cc:460] Webserver started at http://127.8.158.126:44171/ using document root <none> and password file <none>
I20260929 19:03:08.942679 11725 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:08.942749 11725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:08.942895 11725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:08.943953 11725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "e2dd54656f11496fa4d15e5c5c2f355b"
format_stamp: "Formatted at 2026-09-29 19:03:08 on dist-test-slave-nmsz"
server_key: "a589222af2544b40514aca2316b0e5d2"
server_key_iv: "726e8d9a47f5cc4d67f5eaff450e4c18"
server_key_version: "encryptionkey@0"
I20260929 19:03:08.944334 11725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "e2dd54656f11496fa4d15e5c5c2f355b"
format_stamp: "Formatted at 2026-09-29 19:03:08 on dist-test-slave-nmsz"
server_key: "a589222af2544b40514aca2316b0e5d2"
server_key_iv: "726e8d9a47f5cc4d67f5eaff450e4c18"
server_key_version: "encryptionkey@0"
I20260929 19:03:08.945679 11725 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:03:08.946496 11740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:08.946767 11725 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:03:08.946890 11725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "e2dd54656f11496fa4d15e5c5c2f355b"
format_stamp: "Formatted at 2026-09-29 19:03:08 on dist-test-slave-nmsz"
server_key: "a589222af2544b40514aca2316b0e5d2"
server_key_iv: "726e8d9a47f5cc4d67f5eaff450e4c18"
server_key_version: "encryptionkey@0"
I20260929 19:03:08.946978 11725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:08.989104 11725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:08.991536 11725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:08.991727 11725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:08.995522 11725 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.126:34935
I20260929 19:03:08.995556 11792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.126:34935 every 8 connection(s)
I20260929 19:03:08.995918 11725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:03:08.996748 11793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:08.999058 11793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2dd54656f11496fa4d15e5c5c2f355b: Bootstrap starting.
I20260929 19:03:08.999630 11793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2dd54656f11496fa4d15e5c5c2f355b: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:08.999920 11793 log.cc:824] T 00000000000000000000000000000000 P e2dd54656f11496fa4d15e5c5c2f355b: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:09.000522 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 11725
I20260929 19:03:09.000610 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:03:09.000663 11793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2dd54656f11496fa4d15e5c5c2f355b: No bootstrap required, opened a new log
I20260929 19:03:09.000780 8825 external_mini_cluster.cc:1428] Setting key 8fa30800d87e616a7b60e0093c9acff8
I20260929 19:03:09.003120 11793 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2dd54656f11496fa4d15e5c5c2f355b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2dd54656f11496fa4d15e5c5c2f355b" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 34935 } }
I20260929 19:03:09.003273 11793 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2dd54656f11496fa4d15e5c5c2f355b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:09.003316 11793 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2dd54656f11496fa4d15e5c5c2f355b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2dd54656f11496fa4d15e5c5c2f355b, State: Initialized, Role: FOLLOWER
I20260929 19:03:09.003451 11793 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2dd54656f11496fa4d15e5c5c2f355b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2dd54656f11496fa4d15e5c5c2f355b" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 34935 } }
I20260929 19:03:09.003546 11793 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2dd54656f11496fa4d15e5c5c2f355b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:03:09.003592 11793 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2dd54656f11496fa4d15e5c5c2f355b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:03:09.003640 11793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2dd54656f11496fa4d15e5c5c2f355b [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:09.004144 11793 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2dd54656f11496fa4d15e5c5c2f355b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2dd54656f11496fa4d15e5c5c2f355b" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 34935 } }
I20260929 19:03:09.004259 11793 leader_election.cc:302] T 00000000000000000000000000000000 P e2dd54656f11496fa4d15e5c5c2f355b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2dd54656f11496fa4d15e5c5c2f355b; no voters:
I20260929 19:03:09.004426 11793 leader_election.cc:288] T 00000000000000000000000000000000 P e2dd54656f11496fa4d15e5c5c2f355b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:03:09.004465 11798 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2dd54656f11496fa4d15e5c5c2f355b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:03:09.004612 11798 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2dd54656f11496fa4d15e5c5c2f355b [term 1 LEADER]: Becoming Leader. State: Replica: e2dd54656f11496fa4d15e5c5c2f355b, State: Running, Role: LEADER
I20260929 19:03:09.004690 11793 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2dd54656f11496fa4d15e5c5c2f355b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:03:09.004701 11798 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2dd54656f11496fa4d15e5c5c2f355b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2dd54656f11496fa4d15e5c5c2f355b" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 34935 } }
I20260929 19:03:09.005004 11798 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2dd54656f11496fa4d15e5c5c2f355b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2dd54656f11496fa4d15e5c5c2f355b. Latest consensus state: current_term: 1 leader_uuid: "e2dd54656f11496fa4d15e5c5c2f355b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2dd54656f11496fa4d15e5c5c2f355b" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 34935 } } }
I20260929 19:03:09.005084 11798 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2dd54656f11496fa4d15e5c5c2f355b [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:09.005013 11799 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2dd54656f11496fa4d15e5c5c2f355b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2dd54656f11496fa4d15e5c5c2f355b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2dd54656f11496fa4d15e5c5c2f355b" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 34935 } } }
I20260929 19:03:09.005273 11803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:03:09.005278 11799 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2dd54656f11496fa4d15e5c5c2f355b [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:09.005779 11803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:03:09.008787 11803 catalog_manager.cc:1347] Generated new cluster ID: 8965b926132c40a099c5b1cdbca54744
I20260929 19:03:09.008855 11803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 19:03:09 dist-test-slave-nmsz krb5kdc[11712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:03:09.013566 11797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:09.001238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37328 (local address 127.8.158.126:34935)
0929 19:03:09.001412 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:09.001419 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:09.001443 (+ 24us) server_negotiation.cc:407] Connection header received
0929 19:03:09.003140 (+ 1697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:09.003147 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:09.003207 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:09.003399 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:09.004150 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:09.005568 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:09.006693 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:09.007090 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:09.007219 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:09.009933 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:09.009937 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:09.009937 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:09.009948 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:09.011928 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:09.012372 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:09.012375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:09.012375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:09.012492 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:09.012712 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:09.012713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:09.012713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:09.013000 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:09.013139 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:09.013263 (+ 124us) server_negotiation.cc:299] Negotiation successful
0929 19:03:09.013386 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
I20260929 19:03:09.014572 8825 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:03:09.030843 11797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:09.026845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37342 (local address 127.8.158.126:34935)
0929 19:03:09.026907 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:09.026909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:09.027058 (+ 149us) server_negotiation.cc:407] Connection header received
0929 19:03:09.027107 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:09.027108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:09.027119 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:09.027171 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:09.027562 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:09.028258 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:09.029288 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:09.029589 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:09.029648 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:09.030378 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:09.030379 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:09.030380 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:03:09.030591 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:09.030608 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:03:09.030613 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:09.030669 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:09.030744 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 19:03:09.030767 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 19:03:09.032446 11803 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:03:09.033130 11803 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:03:09.041285 11803 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2dd54656f11496fa4d15e5c5c2f355b: Generated new TSK 0
I20260929 19:03:09.041441 11803 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:03:09.045625 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 11725
2026-09-29T19:03:09Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1367 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:03:09 dist-test-slave-nmsz krb5kdc[11826](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:03:09 dist-test-slave-nmsz krb5kdc[11826](info): set up 2 sockets
Sep 29 19:03:09 dist-test-slave-nmsz krb5kdc[11826](info): commencing operation
krb5kdc: starting...
W20260929 19:03:10.082216 8825 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Sep 29 19:03:10 dist-test-slave-nmsz krb5kdc[11826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:03:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:03:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.158.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
I20260929 19:03:10.200790 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:33955
--webserver_interface=127.8.158.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:35779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:33955
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:10.280419 11839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:10.280664 11839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:10.280701 11839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:10.281854 11839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:03:10.281919 11839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:10.281944 11839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:10.281973 11839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:03:10.282006 11839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:03:10.283613 11839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:35779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:33955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:33955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.158.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:10.283867 11839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:10.284097 11839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:10.288975 11845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:10.289099 11847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:10.288975 11844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:10.289244 11839 server_base.cc:1034] running on GCE node
I20260929 19:03:10.289508 11839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:10.289764 11839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:10.290900 11839 hybrid_clock.cc:648] HybridClock initialized: now 1790708590290883 us; error 34 us; skew 500 ppm
Sep 29 19:03:10 dist-test-slave-nmsz krb5kdc[11826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708590, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:10.293386 11839 init.cc:377] Logged in from keytab as kudu/127.8.158.126@KRBTEST.COM (short username kudu)
I20260929 19:03:10.293840 11839 webserver.cc:460] Webserver started at http://127.8.158.126:41629/ using document root <none> and password file <none>
I20260929 19:03:10.294073 11839 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:10.294116 11839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:10.294281 11839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:10.295306 11839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "2e5562c40ddb46eba1b29db69e88a9aa"
format_stamp: "Formatted at 2026-09-29 19:03:10 on dist-test-slave-nmsz"
server_key: "5f778a38792419769bb9d37e3c89b530"
server_key_iv: "be9efff1b1207fe3dd3150ccbfac1e5b"
server_key_version: "encryptionkey@0"
I20260929 19:03:10.295686 11839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "2e5562c40ddb46eba1b29db69e88a9aa"
format_stamp: "Formatted at 2026-09-29 19:03:10 on dist-test-slave-nmsz"
server_key: "5f778a38792419769bb9d37e3c89b530"
server_key_iv: "be9efff1b1207fe3dd3150ccbfac1e5b"
server_key_version: "encryptionkey@0"
I20260929 19:03:10.297060 11839 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:03:10.297799 11854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:10.298008 11839 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:10.298079 11839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "2e5562c40ddb46eba1b29db69e88a9aa"
format_stamp: "Formatted at 2026-09-29 19:03:10 on dist-test-slave-nmsz"
server_key: "5f778a38792419769bb9d37e3c89b530"
server_key_iv: "be9efff1b1207fe3dd3150ccbfac1e5b"
server_key_version: "encryptionkey@0"
I20260929 19:03:10.298159 11839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:10.322034 11839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:10.324350 11839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:10.324527 11839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:10.328706 11839 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.126:33955
I20260929 19:03:10.328766 11906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.126:33955 every 8 connection(s)
I20260929 19:03:10.329123 11839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:03:10.330101 11907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:10.332309 11907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e5562c40ddb46eba1b29db69e88a9aa: Bootstrap starting.
I20260929 19:03:10.332876 11907 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e5562c40ddb46eba1b29db69e88a9aa: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:10.333154 11907 log.cc:824] T 00000000000000000000000000000000 P 2e5562c40ddb46eba1b29db69e88a9aa: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:10.333895 11907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e5562c40ddb46eba1b29db69e88a9aa: No bootstrap required, opened a new log
I20260929 19:03:10.335305 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 11839
I20260929 19:03:10.335386 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:03:10.335553 8825 external_mini_cluster.cc:1428] Setting key 755da012530e335cb193f95416a39f1a
I20260929 19:03:10.335807 11907 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e5562c40ddb46eba1b29db69e88a9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5562c40ddb46eba1b29db69e88a9aa" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 33955 } }
I20260929 19:03:10.336026 11907 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e5562c40ddb46eba1b29db69e88a9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:10.336108 11907 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e5562c40ddb46eba1b29db69e88a9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e5562c40ddb46eba1b29db69e88a9aa, State: Initialized, Role: FOLLOWER
I20260929 19:03:10.336308 11907 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e5562c40ddb46eba1b29db69e88a9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5562c40ddb46eba1b29db69e88a9aa" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 33955 } }
I20260929 19:03:10.336385 11907 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e5562c40ddb46eba1b29db69e88a9aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:03:10.336407 11907 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e5562c40ddb46eba1b29db69e88a9aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:03:10.336481 11907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e5562c40ddb46eba1b29db69e88a9aa [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:10.337589 11907 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e5562c40ddb46eba1b29db69e88a9aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5562c40ddb46eba1b29db69e88a9aa" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 33955 } }
I20260929 19:03:10.337673 11907 leader_election.cc:302] T 00000000000000000000000000000000 P 2e5562c40ddb46eba1b29db69e88a9aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e5562c40ddb46eba1b29db69e88a9aa; no voters:
I20260929 19:03:10.337846 11907 leader_election.cc:288] T 00000000000000000000000000000000 P 2e5562c40ddb46eba1b29db69e88a9aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:03:10.337877 11912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e5562c40ddb46eba1b29db69e88a9aa [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:03:10.337977 11912 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e5562c40ddb46eba1b29db69e88a9aa [term 1 LEADER]: Becoming Leader. State: Replica: 2e5562c40ddb46eba1b29db69e88a9aa, State: Running, Role: LEADER
I20260929 19:03:10.338095 11907 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e5562c40ddb46eba1b29db69e88a9aa [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:03:10.338106 11912 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e5562c40ddb46eba1b29db69e88a9aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5562c40ddb46eba1b29db69e88a9aa" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 33955 } }
I20260929 19:03:10.338526 11912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e5562c40ddb46eba1b29db69e88a9aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e5562c40ddb46eba1b29db69e88a9aa. Latest consensus state: current_term: 1 leader_uuid: "2e5562c40ddb46eba1b29db69e88a9aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5562c40ddb46eba1b29db69e88a9aa" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 33955 } } }
I20260929 19:03:10.338533 11913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e5562c40ddb46eba1b29db69e88a9aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e5562c40ddb46eba1b29db69e88a9aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5562c40ddb46eba1b29db69e88a9aa" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 33955 } } }
I20260929 19:03:10.338696 11912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e5562c40ddb46eba1b29db69e88a9aa [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:10.338707 11913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e5562c40ddb46eba1b29db69e88a9aa [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:10.338943 11921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:03:10.339496 11921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 29 19:03:10 dist-test-slave-nmsz krb5kdc[11826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:03:10.342821 11921 catalog_manager.cc:1347] Generated new cluster ID: a47aa4461a4a43018c13b393bf105e4e
I20260929 19:03:10.342911 11921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:03:10.346215 11911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:10.336063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58096 (local address 127.8.158.126:33955)
0929 19:03:10.336227 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:10.336234 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:10.336260 (+ 26us) server_negotiation.cc:407] Connection header received
0929 19:03:10.336397 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:10.336404 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:10.336460 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:10.336618 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:10.337387 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:10.338655 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:10.339803 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:10.340373 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:10.340484 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:10.343088 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:10.343092 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:10.343092 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:10.343102 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:10.344767 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:10.345141 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:10.345145 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:10.345145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:10.345254 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:10.345467 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:10.345468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:10.345469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:10.345689 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:10.345864 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:10.345930 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 19:03:10.346062 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":89,"threads_started":1}
I20260929 19:03:10.361867 11921 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:03:10.362563 11921 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:03:10.372705 11921 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e5562c40ddb46eba1b29db69e88a9aa: Generated new TSK 0
I20260929 19:03:10.372882 11921 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.158.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab.
Entry for principal kudu/127.8.158.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab.
I20260929 19:03:10.405217 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.65:0
--local_ip_for_outbound_sockets=127.8.158.65
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:33955
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab
--principal=kudu/127.8.158.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:35779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:10.483711 11933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:10.483901 11933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:10.483922 11933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:10.485055 11933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:10.485098 11933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:10.485124 11933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.65
I20260929 19:03:10.486521 11933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:35779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab
--principal=kudu/127.8.158.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:33955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.65
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:10.486734 11933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:10.486974 11933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:10.491866 11941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:10.491870 11938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:10.491916 11939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:10.492245 11933 server_base.cc:1034] running on GCE node
I20260929 19:03:10.492427 11933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:10.492658 11933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:10.493814 11933 hybrid_clock.cc:648] HybridClock initialized: now 1790708590493800 us; error 45 us; skew 500 ppm
Sep 29 19:03:10 dist-test-slave-nmsz krb5kdc[11826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708590, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:10.496114 11933 init.cc:377] Logged in from keytab as kudu/127.8.158.65@KRBTEST.COM (short username kudu)
I20260929 19:03:10.496517 11933 webserver.cc:460] Webserver started at http://127.8.158.65:45967/ using document root <none> and password file <none>
I20260929 19:03:10.496728 11933 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:10.496791 11933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:10.496933 11933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:10.497905 11933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-0/data/instance:
uuid: "7b896aab4661412487629a550db74c70"
format_stamp: "Formatted at 2026-09-29 19:03:10 on dist-test-slave-nmsz"
server_key: "9993b697fb1ebe47db2b9ae465800b5e"
server_key_iv: "d61380a502537c33a8e858ad0a108166"
server_key_version: "encryptionkey@0"
I20260929 19:03:10.498245 11933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance:
uuid: "7b896aab4661412487629a550db74c70"
format_stamp: "Formatted at 2026-09-29 19:03:10 on dist-test-slave-nmsz"
server_key: "9993b697fb1ebe47db2b9ae465800b5e"
server_key_iv: "d61380a502537c33a8e858ad0a108166"
server_key_version: "encryptionkey@0"
I20260929 19:03:10.499568 11933 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:03:10.500336 11948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:10.500584 11933 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:03:10.500700 11933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-0/wal
uuid: "7b896aab4661412487629a550db74c70"
format_stamp: "Formatted at 2026-09-29 19:03:10 on dist-test-slave-nmsz"
server_key: "9993b697fb1ebe47db2b9ae465800b5e"
server_key_iv: "d61380a502537c33a8e858ad0a108166"
server_key_version: "encryptionkey@0"
I20260929 19:03:10.500778 11933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:10.518115 11933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:10.519902 11933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:10.520066 11933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:10.520296 11933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:10.520603 11933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:10.520656 11933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:10.520689 11933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:10.520715 11933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:10.526296 11933 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.65:34851
I20260929 19:03:10.526364 12061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.65:34851 every 8 connection(s)
I20260929 19:03:10.526643 11933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
I20260929 19:03:10.529616 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 11933
I20260929 19:03:10.529733 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance
I20260929 19:03:10.529932 8825 external_mini_cluster.cc:1428] Setting key b3b99cbdd134946df101b0ce4faa2174
Sep 29 19:03:10 dist-test-slave-nmsz krb5kdc[11826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708590, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:03:10.536700 11911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:10.527341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:53643 (local address 127.8.158.126:33955)
0929 19:03:10.527416 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:10.527419 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:10.527603 (+ 184us) server_negotiation.cc:407] Connection header received
0929 19:03:10.528113 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:10.528114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:10.528130 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:10.528192 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:10.529076 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:10.529576 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:10.530889 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:10.531098 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:10.531180 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:10.533813 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:10.533814 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:10.533815 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:10.533821 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:10.535250 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:10.535753 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:10.535754 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:10.535754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:10.535799 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:10.536061 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:10.536062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:10.536062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:10.536157 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:10.536205 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:10.536564 (+ 359us) server_negotiation.cc:299] Negotiation successful
0929 19:03:10.536638 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 19:03:10.536924 12064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:10.527299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:33955 (local address 127.8.158.65:53643)
0929 19:03:10.527477 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:10.527518 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:10.528011 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:10.528260 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:10.528265 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:10.528800 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:10.528976 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:10.528980 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:10.529667 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:10.529668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:10.530766 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:10.530768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:10.531187 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:10.531189 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:10.531307 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:10.531826 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:10.531837 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:10.533727 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:10.535336 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:10.535343 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:10.535346 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:10.535675 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:10.535844 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:10.535845 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:10.535845 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:10.535998 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:10.536226 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:10.536233 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:10.536394 (+ 161us) client_negotiation.cc:769] Sending connection context
0929 19:03:10.536521 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 19:03:10.536682 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":88,"threads_started":1}
I20260929 19:03:10.537415 12062 heartbeater.cc:347] Connected to a master server at 127.8.158.126:33955
I20260929 19:03:10.537526 12062 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:10.537694 12062 heartbeater.cc:511] Master 127.8.158.126:33955 requested a full tablet report, sending...
I20260929 19:03:10.538269 11871 ts_manager.cc:194] Registered new tserver with Master: 7b896aab4661412487629a550db74c70 (127.8.158.65:34851)
WARNING: no policy specified for kudu/127.8.158.66@KRBTEST.COM; defaulting to no policy
I20260929 19:03:10.539104 11871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.65@KRBTEST.COM'} at 127.8.158.65:53643
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab.
Entry for principal kudu/127.8.158.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab.
I20260929 19:03:10.558305 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.66:0
--local_ip_for_outbound_sockets=127.8.158.66
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:33955
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab
--principal=kudu/127.8.158.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:35779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:10.636663 12067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:10.636852 12067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:10.636874 12067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:10.638120 12067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:10.638157 12067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:10.638183 12067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.66
I20260929 19:03:10.639778 12067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:35779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab
--principal=kudu/127.8.158.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:33955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.66
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:10.640017 12067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:10.640235 12067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:10.645184 12072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:10.645277 12073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:10.645180 12067 server_base.cc:1034] running on GCE node
W20260929 19:03:10.645179 12075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:10.645587 12067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:10.645831 12067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:10.646983 12067 hybrid_clock.cc:648] HybridClock initialized: now 1790708590646971 us; error 30 us; skew 500 ppm
Sep 29 19:03:10 dist-test-slave-nmsz krb5kdc[11826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708590, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:10.649392 12067 init.cc:377] Logged in from keytab as kudu/127.8.158.66@KRBTEST.COM (short username kudu)
I20260929 19:03:10.649807 12067 webserver.cc:460] Webserver started at http://127.8.158.66:41699/ using document root <none> and password file <none>
I20260929 19:03:10.650082 12067 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:10.650154 12067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:10.650292 12067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:10.651316 12067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-1/data/instance:
uuid: "8f1497a68b254db58e508a45d275cf90"
format_stamp: "Formatted at 2026-09-29 19:03:10 on dist-test-slave-nmsz"
server_key: "611c49450ee0e404801928e613f71c19"
server_key_iv: "b5d046e88347aeb2ea82dc2be8fb3a91"
server_key_version: "encryptionkey@0"
I20260929 19:03:10.651669 12067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance:
uuid: "8f1497a68b254db58e508a45d275cf90"
format_stamp: "Formatted at 2026-09-29 19:03:10 on dist-test-slave-nmsz"
server_key: "611c49450ee0e404801928e613f71c19"
server_key_iv: "b5d046e88347aeb2ea82dc2be8fb3a91"
server_key_version: "encryptionkey@0"
I20260929 19:03:10.653028 12067 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 19:03:10.653750 12082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:10.653996 12067 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:03:10.654091 12067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-1/wal
uuid: "8f1497a68b254db58e508a45d275cf90"
format_stamp: "Formatted at 2026-09-29 19:03:10 on dist-test-slave-nmsz"
server_key: "611c49450ee0e404801928e613f71c19"
server_key_iv: "b5d046e88347aeb2ea82dc2be8fb3a91"
server_key_version: "encryptionkey@0"
I20260929 19:03:10.654165 12067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:10.671213 12067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:10.673336 12067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:10.673491 12067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:10.673705 12067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:10.674032 12067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:10.674093 12067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:10.674125 12067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:10.674151 12067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:10.679728 12067 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.66:42291
I20260929 19:03:10.679793 12195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.66:42291 every 8 connection(s)
I20260929 19:03:10.680092 12067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
I20260929 19:03:10.682749 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 12067
I20260929 19:03:10.682900 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance
I20260929 19:03:10.683103 8825 external_mini_cluster.cc:1428] Setting key 4b36636f24cace2eaa3302cc39dd3633
Sep 29 19:03:10 dist-test-slave-nmsz krb5kdc[11826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708590, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.66@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:03:10.690078 11911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:10.680791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:43255 (local address 127.8.158.126:33955)
0929 19:03:10.680867 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:10.680870 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:10.681044 (+ 174us) server_negotiation.cc:407] Connection header received
0929 19:03:10.681502 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:10.681503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:10.681513 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:10.681566 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:10.682589 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:10.683152 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:10.684328 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:10.684534 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:10.684609 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:10.687036 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:10.687037 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:10.687037 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:10.687044 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:10.688539 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:10.689065 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:10.689066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:10.689066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:10.689130 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:10.689442 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:10.689442 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:10.689442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:10.689543 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:10.689596 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:10.689949 (+ 353us) server_negotiation.cc:299] Negotiation successful
0929 19:03:10.690016 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 19:03:10.690435 12198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:10.680753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:33955 (local address 127.8.158.66:43255)
0929 19:03:10.680925 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:10.680961 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:10.681419 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:10.681629 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:10.681632 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:10.682209 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:10.682494 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:10.682499 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:10.683293 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:10.683294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:10.684228 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:10.684230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:10.684613 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:10.684614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:10.684741 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:10.685254 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:10.685261 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:10.686927 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:10.688623 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:10.688629 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:10.688632 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:10.688963 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:10.689210 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:10.689211 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:10.689211 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:10.689369 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:10.689609 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:10.689616 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:10.689796 (+ 180us) client_negotiation.cc:769] Sending connection context
0929 19:03:10.689917 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 19:03:10.690082 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":104,"threads_started":1}
I20260929 19:03:10.690943 12196 heartbeater.cc:347] Connected to a master server at 127.8.158.126:33955
I20260929 19:03:10.691042 12196 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:10.691224 12196 heartbeater.cc:511] Master 127.8.158.126:33955 requested a full tablet report, sending...
I20260929 19:03:10.691565 11871 ts_manager.cc:194] Registered new tserver with Master: 8f1497a68b254db58e508a45d275cf90 (127.8.158.66:42291)
I20260929 19:03:10.691879 11871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.66@KRBTEST.COM'} at 127.8.158.66:43255
WARNING: no policy specified for kudu/127.8.158.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab.
Entry for principal kudu/127.8.158.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab.
I20260929 19:03:10.710673 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.67:0
--local_ip_for_outbound_sockets=127.8.158.67
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:33955
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab
--principal=kudu/127.8.158.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:35779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:10.791100 12201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:10.791316 12201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:10.791350 12201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:10.792451 12201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:10.792516 12201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:10.792560 12201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.67
I20260929 19:03:10.793985 12201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:35779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab
--principal=kudu/127.8.158.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:33955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.67
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:10.794237 12201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:10.794474 12201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:10.799113 12206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:10.799160 12207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:10.799130 12209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:10.799371 12201 server_base.cc:1034] running on GCE node
I20260929 19:03:10.799561 12201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:10.799806 12201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:10.800958 12201 hybrid_clock.cc:648] HybridClock initialized: now 1790708590800943 us; error 26 us; skew 500 ppm
Sep 29 19:03:10 dist-test-slave-nmsz krb5kdc[11826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708590, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:10.803500 12201 init.cc:377] Logged in from keytab as kudu/127.8.158.67@KRBTEST.COM (short username kudu)
I20260929 19:03:10.803951 12201 webserver.cc:460] Webserver started at http://127.8.158.67:45203/ using document root <none> and password file <none>
I20260929 19:03:10.804174 12201 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:10.804240 12201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:10.804404 12201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:10.805364 12201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-2/data/instance:
uuid: "8c7579409c8e4f8787a171dc7dbd3aa6"
format_stamp: "Formatted at 2026-09-29 19:03:10 on dist-test-slave-nmsz"
server_key: "25374ed3edadf89ff1f4555daeea9c82"
server_key_iv: "425c24d59ef1649bbfb83a3b84fb0158"
server_key_version: "encryptionkey@0"
I20260929 19:03:10.805706 12201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance:
uuid: "8c7579409c8e4f8787a171dc7dbd3aa6"
format_stamp: "Formatted at 2026-09-29 19:03:10 on dist-test-slave-nmsz"
server_key: "25374ed3edadf89ff1f4555daeea9c82"
server_key_iv: "425c24d59ef1649bbfb83a3b84fb0158"
server_key_version: "encryptionkey@0"
I20260929 19:03:10.807253 12201 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:03:10.808033 12216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:10.808279 12201 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 19:03:10.808404 12201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-2/wal
uuid: "8c7579409c8e4f8787a171dc7dbd3aa6"
format_stamp: "Formatted at 2026-09-29 19:03:10 on dist-test-slave-nmsz"
server_key: "25374ed3edadf89ff1f4555daeea9c82"
server_key_iv: "425c24d59ef1649bbfb83a3b84fb0158"
server_key_version: "encryptionkey@0"
I20260929 19:03:10.808487 12201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:10.822665 12201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:10.824544 12201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:10.824698 12201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:10.824906 12201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:10.825181 12201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:10.825233 12201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:10.825273 12201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:10.825302 12201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:10.831073 12201 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.67:42017
I20260929 19:03:10.831106 12329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.67:42017 every 8 connection(s)
I20260929 19:03:10.831408 12201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
I20260929 19:03:10.834903 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 12201
I20260929 19:03:10.835013 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance
I20260929 19:03:10.835206 8825 external_mini_cluster.cc:1428] Setting key 0f1d64f9c787d2b5dbde7f7784c0b6a8
Sep 29 19:03:10 dist-test-slave-nmsz krb5kdc[11826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708590, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.67@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:03:10.840372 11911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:10.832100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:44869 (local address 127.8.158.126:33955)
0929 19:03:10.832146 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:10.832148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:10.832396 (+ 248us) server_negotiation.cc:407] Connection header received
0929 19:03:10.832805 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:10.832806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:10.832816 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:10.832873 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:10.833743 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:10.834218 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:10.835374 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:10.835565 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:10.835620 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:10.837710 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:10.837711 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:10.837711 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:10.837717 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:10.839008 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:10.839411 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:10.839412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:10.839413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:10.839476 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:10.839738 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:10.839739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:10.839740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:10.839866 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:10.839944 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:10.840199 (+ 255us) server_negotiation.cc:299] Negotiation successful
0929 19:03:10.840274 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 19:03:10.840536 12332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:10.832101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:33955 (local address 127.8.158.67:44869)
0929 19:03:10.832263 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:10.832295 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:10.832725 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:10.832943 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:10.832946 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:10.833470 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:10.833639 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:10.833644 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:10.834296 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:10.834297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:10.835280 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:10.835283 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:10.835665 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:10.835666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:10.835810 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:10.836330 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:10.836337 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:10.837635 (+ 1298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:10.839080 (+ 1445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:10.839085 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:10.839087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:10.839281 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:10.839543 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:10.839544 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:10.839545 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:10.839643 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:10.839935 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:10.839941 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:10.840069 (+ 128us) client_negotiation.cc:769] Sending connection context
0929 19:03:10.840177 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 19:03:10.840320 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":80,"threads_started":1}
I20260929 19:03:10.841034 12330 heartbeater.cc:347] Connected to a master server at 127.8.158.126:33955
I20260929 19:03:10.841107 12330 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:10.841287 12330 heartbeater.cc:511] Master 127.8.158.126:33955 requested a full tablet report, sending...
I20260929 19:03:10.841661 11871 ts_manager.cc:194] Registered new tserver with Master: 8c7579409c8e4f8787a171dc7dbd3aa6 (127.8.158.67:42017)
I20260929 19:03:10.842031 11871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.67@KRBTEST.COM'} at 127.8.158.67:44869
I20260929 19:03:10.846889 8825 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 19:03:10 dist-test-slave-nmsz krb5kdc[11826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708590, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 19:03:10 dist-test-slave-nmsz krb5kdc[11826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708590, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:03:10.868795 11911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:10.861769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58104 (local address 127.8.158.126:33955)
0929 19:03:10.861859 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:10.861861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:10.861991 (+ 130us) server_negotiation.cc:407] Connection header received
0929 19:03:10.862092 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:10.862093 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:10.862103 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:10.862165 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:10.862800 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:10.863297 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:10.864039 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:10.864222 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:10.864304 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:10.866486 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:10.866488 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:10.866488 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:10.866494 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:10.867895 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:10.868194 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:10.868195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:10.868195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:10.868241 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:10.868403 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:10.868403 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:10.868403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:10.868528 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:10.868597 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:10.868684 (+ 87us) server_negotiation.cc:299] Negotiation successful
0929 19:03:10.868736 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260929 19:03:10.869081 11871 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58104
I20260929 19:03:10.869573 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 11933
I20260929 19:03:10.874557 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 12067
I20260929 19:03:10.878947 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 12201
I20260929 19:03:10.882896 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 11839
2026-09-29T19:03:10Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1839 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:03:10 dist-test-slave-nmsz krb5kdc[12342](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:03:10 dist-test-slave-nmsz krb5kdc[12342](info): set up 2 sockets
Sep 29 19:03:10 dist-test-slave-nmsz krb5kdc[12342](info): commencing operation
krb5kdc: starting...
W20260929 19:03:11.918516 8825 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Sep 29 19:03:12 dist-test-slave-nmsz krb5kdc[12342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:03:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:03:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.158.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
I20260929 19:03:12.041446 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:40679
--webserver_interface=127.8.158.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:43647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:40679
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:12.120980 12355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:12.121197 12355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:12.121220 12355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:12.122421 12355 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:03:12.122459 12355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:12.122473 12355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:12.122488 12355 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:03:12.122499 12355 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:03:12.123997 12355 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:43647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:40679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:40679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.158.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:12.124238 12355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:12.124471 12355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:12.130308 12361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:12.130374 12363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:12.130503 12360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:12.130796 12355 server_base.cc:1034] running on GCE node
I20260929 19:03:12.131031 12355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:12.131281 12355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:12.132459 12355 hybrid_clock.cc:648] HybridClock initialized: now 1790708592132433 us; error 55 us; skew 500 ppm
Sep 29 19:03:12 dist-test-slave-nmsz krb5kdc[12342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708592, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:12.135111 12355 init.cc:377] Logged in from keytab as kudu/127.8.158.126@KRBTEST.COM (short username kudu)
I20260929 19:03:12.135598 12355 webserver.cc:460] Webserver started at http://127.8.158.126:38645/ using document root <none> and password file <none>
I20260929 19:03:12.135824 12355 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:12.135893 12355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:12.136045 12355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:12.137106 12355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "5d7730ebdfac413c967e0736376d5818"
format_stamp: "Formatted at 2026-09-29 19:03:12 on dist-test-slave-nmsz"
server_key: "3c7714ec17b6191174fa1a1e643b0db6"
server_key_iv: "fbd5cea96dfaa2253f70ddab68944536"
server_key_version: "encryptionkey@0"
I20260929 19:03:12.137463 12355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "5d7730ebdfac413c967e0736376d5818"
format_stamp: "Formatted at 2026-09-29 19:03:12 on dist-test-slave-nmsz"
server_key: "3c7714ec17b6191174fa1a1e643b0db6"
server_key_iv: "fbd5cea96dfaa2253f70ddab68944536"
server_key_version: "encryptionkey@0"
I20260929 19:03:12.138888 12355 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:03:12.139673 12370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:12.139868 12355 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:12.139969 12355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "5d7730ebdfac413c967e0736376d5818"
format_stamp: "Formatted at 2026-09-29 19:03:12 on dist-test-slave-nmsz"
server_key: "3c7714ec17b6191174fa1a1e643b0db6"
server_key_iv: "fbd5cea96dfaa2253f70ddab68944536"
server_key_version: "encryptionkey@0"
I20260929 19:03:12.140050 12355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:12.154757 12355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:12.156742 12355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:12.156911 12355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:12.160894 12355 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.126:40679
I20260929 19:03:12.160943 12422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.126:40679 every 8 connection(s)
I20260929 19:03:12.161444 12355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:03:12.162070 12423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:12.164206 12423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d7730ebdfac413c967e0736376d5818: Bootstrap starting.
I20260929 19:03:12.164774 12423 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d7730ebdfac413c967e0736376d5818: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:12.165063 12423 log.cc:824] T 00000000000000000000000000000000 P 5d7730ebdfac413c967e0736376d5818: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:12.165779 12423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d7730ebdfac413c967e0736376d5818: No bootstrap required, opened a new log
I20260929 19:03:12.165783 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 12355
I20260929 19:03:12.165915 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:03:12.166097 8825 external_mini_cluster.cc:1428] Setting key 165d3ec63d9c333b5ed030344e11279c
I20260929 19:03:12.168700 12423 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d7730ebdfac413c967e0736376d5818 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d7730ebdfac413c967e0736376d5818" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 40679 } }
I20260929 19:03:12.168861 12423 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d7730ebdfac413c967e0736376d5818 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:12.168893 12423 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d7730ebdfac413c967e0736376d5818 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d7730ebdfac413c967e0736376d5818, State: Initialized, Role: FOLLOWER
I20260929 19:03:12.169006 12423 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d7730ebdfac413c967e0736376d5818 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d7730ebdfac413c967e0736376d5818" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 40679 } }
I20260929 19:03:12.169073 12423 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d7730ebdfac413c967e0736376d5818 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:03:12.169100 12423 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d7730ebdfac413c967e0736376d5818 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:03:12.169142 12423 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d7730ebdfac413c967e0736376d5818 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:12.169751 12423 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d7730ebdfac413c967e0736376d5818 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d7730ebdfac413c967e0736376d5818" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 40679 } }
I20260929 19:03:12.169876 12423 leader_election.cc:302] T 00000000000000000000000000000000 P 5d7730ebdfac413c967e0736376d5818 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d7730ebdfac413c967e0736376d5818; no voters:
I20260929 19:03:12.170073 12423 leader_election.cc:288] T 00000000000000000000000000000000 P 5d7730ebdfac413c967e0736376d5818 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:03:12.170100 12428 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d7730ebdfac413c967e0736376d5818 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:03:12.170209 12428 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d7730ebdfac413c967e0736376d5818 [term 1 LEADER]: Becoming Leader. State: Replica: 5d7730ebdfac413c967e0736376d5818, State: Running, Role: LEADER
I20260929 19:03:12.170341 12423 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d7730ebdfac413c967e0736376d5818 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:03:12.170289 12428 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d7730ebdfac413c967e0736376d5818 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d7730ebdfac413c967e0736376d5818" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 40679 } }
I20260929 19:03:12.170787 12430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d7730ebdfac413c967e0736376d5818 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d7730ebdfac413c967e0736376d5818" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d7730ebdfac413c967e0736376d5818" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 40679 } } }
I20260929 19:03:12.170874 12430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d7730ebdfac413c967e0736376d5818 [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:12.171103 12429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d7730ebdfac413c967e0736376d5818 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d7730ebdfac413c967e0736376d5818. Latest consensus state: current_term: 1 leader_uuid: "5d7730ebdfac413c967e0736376d5818" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d7730ebdfac413c967e0736376d5818" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 40679 } } }
I20260929 19:03:12.171232 12429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d7730ebdfac413c967e0736376d5818 [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:12.171411 12436 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:03:12.171898 12436 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 29 19:03:12 dist-test-slave-nmsz krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:03:12.173524 12436 catalog_manager.cc:1347] Generated new cluster ID: 36066ec39d87429ca018b73d4defbc19
I20260929 19:03:12.173570 12436 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:03:12.177827 12427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:12.166598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55394 (local address 127.8.158.126:40679)
0929 19:03:12.166777 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:12.166786 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:12.166817 (+ 31us) server_negotiation.cc:407] Connection header received
0929 19:03:12.167264 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:12.167272 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:12.167330 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:12.167484 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:12.168058 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:12.169428 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:12.170298 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:12.170920 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:12.171009 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:12.173941 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:12.173944 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:12.173945 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:12.173956 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:12.176056 (+ 2100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:12.176470 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:12.176473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:12.176473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:12.176571 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:12.176785 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:12.176786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:12.176786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:12.177088 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:12.177216 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:12.177486 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 19:03:12.177622 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":75,"threads_started":1}
I20260929 19:03:12.188302 12436 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:03:12.188789 12436 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:03:12.197634 12436 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d7730ebdfac413c967e0736376d5818: Generated new TSK 0
I20260929 19:03:12.197849 12436 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.158.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab.
Entry for principal kudu/127.8.158.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab.
I20260929 19:03:12.224644 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.65:0
--local_ip_for_outbound_sockets=127.8.158.65
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:40679
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab
--principal=kudu/127.8.158.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:43647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:12.304562 12449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:12.304746 12449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:12.304769 12449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:12.305881 12449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:12.305920 12449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:12.305946 12449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.65
I20260929 19:03:12.307456 12449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:43647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab
--principal=kudu/127.8.158.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:40679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.65
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:12.307678 12449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:12.307946 12449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:12.313248 12457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:12.313117 12454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:12.313126 12455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:12.313490 12449 server_base.cc:1034] running on GCE node
I20260929 19:03:12.313756 12449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:12.314050 12449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:12.315181 12449 hybrid_clock.cc:648] HybridClock initialized: now 1790708592315135 us; error 54 us; skew 500 ppm
Sep 29 19:03:12 dist-test-slave-nmsz krb5kdc[12342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708592, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:12.317828 12449 init.cc:377] Logged in from keytab as kudu/127.8.158.65@KRBTEST.COM (short username kudu)
I20260929 19:03:12.318252 12449 webserver.cc:460] Webserver started at http://127.8.158.65:33205/ using document root <none> and password file <none>
I20260929 19:03:12.318495 12449 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:12.318540 12449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:12.318624 12449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:12.319576 12449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-0/data/instance:
uuid: "ecd397023ef04e35a0db03e9b3374ed5"
format_stamp: "Formatted at 2026-09-29 19:03:12 on dist-test-slave-nmsz"
server_key: "7d05ecf805fcbf249420fa29a3711245"
server_key_iv: "cf4c43d849266b07eef0f041aa064174"
server_key_version: "encryptionkey@0"
I20260929 19:03:12.319865 12449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance:
uuid: "ecd397023ef04e35a0db03e9b3374ed5"
format_stamp: "Formatted at 2026-09-29 19:03:12 on dist-test-slave-nmsz"
server_key: "7d05ecf805fcbf249420fa29a3711245"
server_key_iv: "cf4c43d849266b07eef0f041aa064174"
server_key_version: "encryptionkey@0"
I20260929 19:03:12.321074 12449 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:03:12.321719 12464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:12.321892 12449 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:03:12.321969 12449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-0/wal
uuid: "ecd397023ef04e35a0db03e9b3374ed5"
format_stamp: "Formatted at 2026-09-29 19:03:12 on dist-test-slave-nmsz"
server_key: "7d05ecf805fcbf249420fa29a3711245"
server_key_iv: "cf4c43d849266b07eef0f041aa064174"
server_key_version: "encryptionkey@0"
I20260929 19:03:12.322021 12449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:12.345098 12449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:12.347363 12449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:12.347469 12449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:12.347645 12449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:12.347960 12449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:12.347993 12449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:12.348014 12449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:12.348063 12449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:12.353567 12449 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.65:32887
I20260929 19:03:12.353663 12577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.65:32887 every 8 connection(s)
I20260929 19:03:12.353942 12449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
I20260929 19:03:12.359582 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 12449
Sep 29 19:03:12 dist-test-slave-nmsz krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708592, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:03:12.359694 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance
I20260929 19:03:12.359876 8825 external_mini_cluster.cc:1428] Setting key 572fc6d22fd6950ebe0ad003895b386f
I20260929 19:03:12.362969 12580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:12.354617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40679 (local address 127.8.158.65:55729)
0929 19:03:12.354756 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:12.354798 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:12.355677 (+ 879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:12.355882 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:12.355885 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:12.356131 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:12.356307 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:12.356311 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:12.356945 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:12.356945 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:12.357727 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:12.357729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:12.358061 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:12.358062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:12.358167 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:12.358594 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:12.358602 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:12.360120 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:12.361669 (+ 1549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:12.361676 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:12.361678 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:12.361946 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:12.362150 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:12.362150 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:12.362151 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:12.362251 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:12.362530 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:12.362534 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:12.362636 (+ 102us) client_negotiation.cc:769] Sending connection context
0929 19:03:12.362720 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 19:03:12.362826 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":60,"threads_started":1}
I20260929 19:03:12.362991 12427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:12.354713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:55729 (local address 127.8.158.126:40679)
0929 19:03:12.354784 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:12.354786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:12.354873 (+ 87us) server_negotiation.cc:407] Connection header received
0929 19:03:12.355767 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:12.355768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:12.355778 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:12.355829 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:12.356384 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:12.356870 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:12.357801 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:12.357991 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:12.358044 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:12.360233 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:12.360235 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:12.360235 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:12.360242 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:12.361562 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:12.362017 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:12.362018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:12.362019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:12.362082 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:12.362311 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:12.362312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:12.362312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:12.362466 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:12.362536 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:12.362766 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 19:03:12.362855 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 19:03:12.363402 12578 heartbeater.cc:347] Connected to a master server at 127.8.158.126:40679
I20260929 19:03:12.363492 12578 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:12.363613 12578 heartbeater.cc:511] Master 127.8.158.126:40679 requested a full tablet report, sending...
I20260929 19:03:12.364073 12387 ts_manager.cc:194] Registered new tserver with Master: ecd397023ef04e35a0db03e9b3374ed5 (127.8.158.65:32887)
I20260929 19:03:12.364998 12387 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.65@KRBTEST.COM'} at 127.8.158.65:55729
WARNING: no policy specified for kudu/127.8.158.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab.
Entry for principal kudu/127.8.158.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab.
I20260929 19:03:12.385881 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.66:0
--local_ip_for_outbound_sockets=127.8.158.66
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:40679
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab
--principal=kudu/127.8.158.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:43647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:12.463645 12583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:12.463862 12583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:12.463897 12583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:12.465020 12583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:12.465086 12583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:12.465131 12583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.66
I20260929 19:03:12.466645 12583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:43647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab
--principal=kudu/127.8.158.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:40679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.66
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:12.466892 12583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:12.467103 12583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:12.471880 12589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:12.471900 12591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:12.471880 12588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:12.472015 12583 server_base.cc:1034] running on GCE node
I20260929 19:03:12.472275 12583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:12.472500 12583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:12.473636 12583 hybrid_clock.cc:648] HybridClock initialized: now 1790708592473607 us; error 46 us; skew 500 ppm
Sep 29 19:03:12 dist-test-slave-nmsz krb5kdc[12342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708592, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:12.476131 12583 init.cc:377] Logged in from keytab as kudu/127.8.158.66@KRBTEST.COM (short username kudu)
I20260929 19:03:12.476567 12583 webserver.cc:460] Webserver started at http://127.8.158.66:45365/ using document root <none> and password file <none>
I20260929 19:03:12.476794 12583 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:12.476869 12583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:12.477008 12583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:12.477985 12583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-1/data/instance:
uuid: "ab29316dc025417cb2b58b553cd61c5e"
format_stamp: "Formatted at 2026-09-29 19:03:12 on dist-test-slave-nmsz"
server_key: "c95df1be04fab2c62efeb9ab04ec66f6"
server_key_iv: "172c4953081491e01ffde3ec43c88dc6"
server_key_version: "encryptionkey@0"
I20260929 19:03:12.478381 12583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance:
uuid: "ab29316dc025417cb2b58b553cd61c5e"
format_stamp: "Formatted at 2026-09-29 19:03:12 on dist-test-slave-nmsz"
server_key: "c95df1be04fab2c62efeb9ab04ec66f6"
server_key_iv: "172c4953081491e01ffde3ec43c88dc6"
server_key_version: "encryptionkey@0"
I20260929 19:03:12.479676 12583 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:12.480465 12598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:12.480640 12583 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:12.480726 12583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-1/wal
uuid: "ab29316dc025417cb2b58b553cd61c5e"
format_stamp: "Formatted at 2026-09-29 19:03:12 on dist-test-slave-nmsz"
server_key: "c95df1be04fab2c62efeb9ab04ec66f6"
server_key_iv: "172c4953081491e01ffde3ec43c88dc6"
server_key_version: "encryptionkey@0"
I20260929 19:03:12.480806 12583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:12.500754 12583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:12.502810 12583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:12.503000 12583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:12.503228 12583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:12.503556 12583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:12.503609 12583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:12.503644 12583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:12.503671 12583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:12.509056 12583 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.66:34153
I20260929 19:03:12.509097 12711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.66:34153 every 8 connection(s)
I20260929 19:03:12.509433 12583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
I20260929 19:03:12.510134 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 12583
I20260929 19:03:12.510245 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance
I20260929 19:03:12.510488 8825 external_mini_cluster.cc:1428] Setting key e377db942ed098ec04d493812ec64cdc
Sep 29 19:03:12 dist-test-slave-nmsz krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708592, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.66@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.158.67@KRBTEST.COM; defaulting to no policy
I20260929 19:03:12.519137 12427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:12.510199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:51009 (local address 127.8.158.126:40679)
0929 19:03:12.510248 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:12.510250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:12.510486 (+ 236us) server_negotiation.cc:407] Connection header received
0929 19:03:12.511600 (+ 1114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:12.511601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:12.511613 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:12.511667 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:12.512267 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:12.512757 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:12.513717 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:12.513929 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:12.514026 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:12.516073 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:12.516075 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:12.516075 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:12.516082 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:12.517667 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:12.518104 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:12.518105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:12.518106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:12.518167 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:12.518489 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:12.518490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:12.518490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:12.518630 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:12.518698 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:12.518969 (+ 271us) server_negotiation.cc:299] Negotiation successful
0929 19:03:12.519046 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 19:03:12.519313 12714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:12.510177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40679 (local address 127.8.158.66:51009)
0929 19:03:12.510358 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:12.510405 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:12.511498 (+ 1093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:12.511736 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:12.511740 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:12.511997 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:12.512171 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:12.512175 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:12.512850 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:12.512851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:12.513627 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:12.513629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:12.514035 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:12.514036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:12.514164 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:12.514691 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:12.514699 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:12.515981 (+ 1282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:12.517770 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:12.517777 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:12.517779 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:12.518006 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:12.518233 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:12.518234 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:12.518234 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:12.518419 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:12.518686 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:12.518692 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:12.518830 (+ 138us) client_negotiation.cc:769] Sending connection context
0929 19:03:12.518936 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 19:03:12.519089 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":89,"threads_started":1}
I20260929 19:03:12.519904 12712 heartbeater.cc:347] Connected to a master server at 127.8.158.126:40679
I20260929 19:03:12.519987 12712 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:12.520174 12712 heartbeater.cc:511] Master 127.8.158.126:40679 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.67@KRBTEST.COM" created.
I20260929 19:03:12.520713 12387 ts_manager.cc:194] Registered new tserver with Master: ab29316dc025417cb2b58b553cd61c5e (127.8.158.66:34153)
I20260929 19:03:12.521297 12387 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.66@KRBTEST.COM'} at 127.8.158.66:51009
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab.
Entry for principal kudu/127.8.158.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab.
I20260929 19:03:12.538039 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.67:0
--local_ip_for_outbound_sockets=127.8.158.67
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:40679
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab
--principal=kudu/127.8.158.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:43647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:12.618937 12717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:12.619127 12717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:12.619148 12717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:12.620211 12717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:12.620245 12717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:12.620271 12717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.67
I20260929 19:03:12.621639 12717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:43647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab
--principal=kudu/127.8.158.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:40679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.67
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:12.621810 12717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:12.621969 12717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:12.626770 12723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:12.626797 12725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:12.626758 12722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:12.627022 12717 server_base.cc:1034] running on GCE node
I20260929 19:03:12.627233 12717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:12.627460 12717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:12.628630 12717 hybrid_clock.cc:648] HybridClock initialized: now 1790708592628601 us; error 38 us; skew 500 ppm
Sep 29 19:03:12 dist-test-slave-nmsz krb5kdc[12342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708592, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:12.631157 12717 init.cc:377] Logged in from keytab as kudu/127.8.158.67@KRBTEST.COM (short username kudu)
I20260929 19:03:12.631605 12717 webserver.cc:460] Webserver started at http://127.8.158.67:33847/ using document root <none> and password file <none>
I20260929 19:03:12.631830 12717 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:12.631896 12717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:12.632031 12717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:12.633056 12717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-2/data/instance:
uuid: "75c3febe819a41f298cdd053e5e7bbf1"
format_stamp: "Formatted at 2026-09-29 19:03:12 on dist-test-slave-nmsz"
server_key: "3b0c7f4920f3226ede9162c7ef857ed6"
server_key_iv: "938837b009ce4a062c947fb816e9af3f"
server_key_version: "encryptionkey@0"
I20260929 19:03:12.633419 12717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance:
uuid: "75c3febe819a41f298cdd053e5e7bbf1"
format_stamp: "Formatted at 2026-09-29 19:03:12 on dist-test-slave-nmsz"
server_key: "3b0c7f4920f3226ede9162c7ef857ed6"
server_key_iv: "938837b009ce4a062c947fb816e9af3f"
server_key_version: "encryptionkey@0"
I20260929 19:03:12.634819 12717 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260929 19:03:12.635586 12732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:12.635864 12717 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:12.635969 12717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-2/wal
uuid: "75c3febe819a41f298cdd053e5e7bbf1"
format_stamp: "Formatted at 2026-09-29 19:03:12 on dist-test-slave-nmsz"
server_key: "3b0c7f4920f3226ede9162c7ef857ed6"
server_key_iv: "938837b009ce4a062c947fb816e9af3f"
server_key_version: "encryptionkey@0"
I20260929 19:03:12.636054 12717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:12.645656 12717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:12.647552 12717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:12.647708 12717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:12.647923 12717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:12.648203 12717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:12.648255 12717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:12.648294 12717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:12.648322 12717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:12.654076 12717 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.67:35995
I20260929 19:03:12.654127 12845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.67:35995 every 8 connection(s)
I20260929 19:03:12.654472 12717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
Sep 29 19:03:12 dist-test-slave-nmsz krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708592, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.67@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:03:12.662730 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 12717
I20260929 19:03:12.662837 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance
I20260929 19:03:12.663022 8825 external_mini_cluster.cc:1428] Setting key 112655630ad90844f4bb48edc5af54fc
I20260929 19:03:12.664034 12427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:12.655166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:38933 (local address 127.8.158.126:40679)
0929 19:03:12.655252 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:12.655255 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:12.655449 (+ 194us) server_negotiation.cc:407] Connection header received
0929 19:03:12.656382 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:12.656383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:12.656395 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:12.656453 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:12.657084 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:12.657583 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:12.658579 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:12.658796 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:12.658864 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:12.661081 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:12.661082 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:12.661083 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:12.661091 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:12.662574 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:12.663004 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:12.663006 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:12.663006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:12.663066 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:12.663324 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:12.663324 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:12.663325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:12.663477 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:12.663534 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:12.663870 (+ 336us) server_negotiation.cc:299] Negotiation successful
0929 19:03:12.663944 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 19:03:12.664228 12848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:12.655085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40679 (local address 127.8.158.67:38933)
0929 19:03:12.655303 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:12.655355 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:12.656268 (+ 913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:12.656516 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:12.656520 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:12.656793 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:12.656968 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:12.656973 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:12.657675 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:12.657676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:12.658482 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:12.658485 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:12.658895 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:12.658896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:12.659011 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:12.659576 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:12.659587 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:12.660985 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:12.662667 (+ 1682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:12.662673 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:12.662675 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:12.662945 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:12.663142 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:12.663143 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:12.663144 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:12.663261 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:12.663537 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:12.663544 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:12.663728 (+ 184us) client_negotiation.cc:769] Sending connection context
0929 19:03:12.663850 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 19:03:12.663997 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":105,"threads_started":1}
I20260929 19:03:12.664587 12846 heartbeater.cc:347] Connected to a master server at 127.8.158.126:40679
I20260929 19:03:12.664698 12846 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:12.664860 12846 heartbeater.cc:511] Master 127.8.158.126:40679 requested a full tablet report, sending...
I20260929 19:03:12.665238 12387 ts_manager.cc:194] Registered new tserver with Master: 75c3febe819a41f298cdd053e5e7bbf1 (127.8.158.67:35995)
I20260929 19:03:12.665673 12387 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.67@KRBTEST.COM'} at 127.8.158.67:38933
I20260929 19:03:12.674721 8825 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 19:03:12 dist-test-slave-nmsz krb5kdc[12342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 19:03:12 dist-test-slave-nmsz krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.158.65@KRBTEST.COM
I20260929 19:03:12.696943 12853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:12.689364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58544 (local address 127.8.158.65:32887)
0929 19:03:12.689522 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:12.689526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:12.689536 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:03:12.689559 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:12.689561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:12.689574 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:12.689638 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:12.690195 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:12.690972 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:12.691680 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:12.692023 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:12.692095 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:12.694310 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:12.694351 (+ 41us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:12.694351 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:12.694364 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:12.695952 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:12.696277 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:12.696280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:12.696280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:12.696330 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:12.696506 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:12.696506 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:12.696506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:12.696622 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:12.696718 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:12.696781 (+ 63us) server_negotiation.cc:299] Negotiation successful
0929 19:03:12.696866 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":84,"threads_started":1}
Sep 29 19:03:12 dist-test-slave-nmsz krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:03:12.704841 12427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:12.698268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55404 (local address 127.8.158.126:40679)
0929 19:03:12.698358 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:12.698360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:12.698369 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:03:12.698427 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:12.698428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:12.698436 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:12.698487 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:12.698985 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:12.699441 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:12.700075 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:12.700243 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:12.700291 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:12.702239 (+ 1948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:12.702241 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:12.702241 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:12.702248 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:12.703688 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:12.704054 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:12.704055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:12.704055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:12.704113 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:12.704309 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:12.704309 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:12.704310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:12.704456 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:12.704572 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:12.704673 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 19:03:12.704741 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260929 19:03:12.705030 12387 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55404
I20260929 19:03:12.705354 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 12449
I20260929 19:03:12.709728 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 12583
I20260929 19:03:12.713835 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 12717
I20260929 19:03:12.717789 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 12355
2026-09-29T19:03:12Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1834 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:03:12 dist-test-slave-nmsz krb5kdc[12858](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:03:12 dist-test-slave-nmsz krb5kdc[12858](info): set up 2 sockets
Sep 29 19:03:12 dist-test-slave-nmsz krb5kdc[12858](info): commencing operation
krb5kdc: starting...
W20260929 19:03:13.750252 8825 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Sep 29 19:03:13 dist-test-slave-nmsz krb5kdc[12858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:03:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:03:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.158.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
I20260929 19:03:13.873255 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:43067
--webserver_interface=127.8.158.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:32991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:43067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:13.965628 12871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:13.965831 12871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:13.965853 12871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:13.967061 12871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:03:13.967104 12871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:13.967119 12871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:13.967132 12871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:03:13.967144 12871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:03:13.968648 12871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:32991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:43067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:43067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.158.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:13.968842 12871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:13.969002 12871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:13.974114 12879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:13.974097 12876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:13.974241 12877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:13.974397 12871 server_base.cc:1034] running on GCE node
I20260929 19:03:13.974828 12871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:13.975229 12871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:13.976408 12871 hybrid_clock.cc:648] HybridClock initialized: now 1790708593976382 us; error 43 us; skew 500 ppm
Sep 29 19:03:13 dist-test-slave-nmsz krb5kdc[12858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708593, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:13.983165 12871 init.cc:377] Logged in from keytab as kudu/127.8.158.126@KRBTEST.COM (short username kudu)
I20260929 19:03:13.983518 12871 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:13.983566 12871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:13.983698 12871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:13.984757 12871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "63bb191167ed46a992890d14681572cd"
format_stamp: "Formatted at 2026-09-29 19:03:13 on dist-test-slave-nmsz"
server_key: "744b2ca8caf552e744ff2d49887bef8e"
server_key_iv: "137b1f88751047a6d18a9592e478bbfe"
server_key_version: "encryptionkey@0"
I20260929 19:03:13.985105 12871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "63bb191167ed46a992890d14681572cd"
format_stamp: "Formatted at 2026-09-29 19:03:13 on dist-test-slave-nmsz"
server_key: "744b2ca8caf552e744ff2d49887bef8e"
server_key_iv: "137b1f88751047a6d18a9592e478bbfe"
server_key_version: "encryptionkey@0"
I20260929 19:03:13.986799 12871 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:03:13.987902 12885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:13.988083 12871 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:03:13.988206 12871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "63bb191167ed46a992890d14681572cd"
format_stamp: "Formatted at 2026-09-29 19:03:13 on dist-test-slave-nmsz"
server_key: "744b2ca8caf552e744ff2d49887bef8e"
server_key_iv: "137b1f88751047a6d18a9592e478bbfe"
server_key_version: "encryptionkey@0"
I20260929 19:03:13.988288 12871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:14.004592 12871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:14.007601 12871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:14.007762 12871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:14.012501 12871 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.126:43067
I20260929 19:03:14.012584 12937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.126:43067 every 8 connection(s)
I20260929 19:03:14.012869 12871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:03:14.013824 12938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:14.016121 12938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63bb191167ed46a992890d14681572cd: Bootstrap starting.
I20260929 19:03:14.016695 12938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63bb191167ed46a992890d14681572cd: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:14.016980 12938 log.cc:824] T 00000000000000000000000000000000 P 63bb191167ed46a992890d14681572cd: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:14.017630 12938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63bb191167ed46a992890d14681572cd: No bootstrap required, opened a new log
I20260929 19:03:14.018476 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 12871
I20260929 19:03:14.018582 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:03:14.018764 8825 external_mini_cluster.cc:1428] Setting key 5e610682e0df78cd6ed50763a251c5a4
I20260929 19:03:14.019690 12938 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63bb191167ed46a992890d14681572cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63bb191167ed46a992890d14681572cd" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 43067 } }
I20260929 19:03:14.019923 12938 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63bb191167ed46a992890d14681572cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:14.020004 12938 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63bb191167ed46a992890d14681572cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63bb191167ed46a992890d14681572cd, State: Initialized, Role: FOLLOWER
I20260929 19:03:14.020171 12938 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63bb191167ed46a992890d14681572cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63bb191167ed46a992890d14681572cd" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 43067 } }
I20260929 19:03:14.020264 12938 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63bb191167ed46a992890d14681572cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:03:14.020303 12938 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63bb191167ed46a992890d14681572cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:03:14.020356 12938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63bb191167ed46a992890d14681572cd [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:14.020957 12938 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63bb191167ed46a992890d14681572cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63bb191167ed46a992890d14681572cd" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 43067 } }
I20260929 19:03:14.021075 12938 leader_election.cc:302] T 00000000000000000000000000000000 P 63bb191167ed46a992890d14681572cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63bb191167ed46a992890d14681572cd; no voters:
I20260929 19:03:14.021265 12938 leader_election.cc:288] T 00000000000000000000000000000000 P 63bb191167ed46a992890d14681572cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:03:14.021294 12943 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63bb191167ed46a992890d14681572cd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:03:14.021437 12943 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63bb191167ed46a992890d14681572cd [term 1 LEADER]: Becoming Leader. State: Replica: 63bb191167ed46a992890d14681572cd, State: Running, Role: LEADER
I20260929 19:03:14.021569 12938 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63bb191167ed46a992890d14681572cd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:03:14.021584 12943 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63bb191167ed46a992890d14681572cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63bb191167ed46a992890d14681572cd" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 43067 } }
I20260929 19:03:14.021997 12943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63bb191167ed46a992890d14681572cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63bb191167ed46a992890d14681572cd. Latest consensus state: current_term: 1 leader_uuid: "63bb191167ed46a992890d14681572cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63bb191167ed46a992890d14681572cd" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 43067 } } }
I20260929 19:03:14.022264 12943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63bb191167ed46a992890d14681572cd [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:14.023159 12944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63bb191167ed46a992890d14681572cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63bb191167ed46a992890d14681572cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63bb191167ed46a992890d14681572cd" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 43067 } } }
I20260929 19:03:14.023237 12944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63bb191167ed46a992890d14681572cd [sys.catalog]: This master's current role is: LEADER
W20260929 19:03:14.023443 12959 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 63bb191167ed46a992890d14681572cd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 19:03:14.023499 12959 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:03:14.023551 12952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:03:14.023699 12952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:03:14.025038 12952 catalog_manager.cc:1347] Generated new cluster ID: a49ce9f75bb24d9aad776f58a7b05a9c
I20260929 19:03:14.025089 12952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 19:03:14 dist-test-slave-nmsz krb5kdc[12858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:03:14.029731 12942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:14.019311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47510 (local address 127.8.158.126:43067)
0929 19:03:14.019490 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:14.019497 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:14.019521 (+ 24us) server_negotiation.cc:407] Connection header received
0929 19:03:14.019652 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:14.019657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:14.019698 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:14.019804 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:14.020602 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:14.021757 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:14.022886 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:14.023477 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:14.023610 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:14.026436 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:14.026440 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:14.026440 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:14.026452 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:14.028226 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:14.028620 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:14.028623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:14.028624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:14.028724 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:14.028936 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:14.028936 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:14.028937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:14.029172 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:14.029341 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:14.029421 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 19:03:14.029554 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":98,"threads_started":1}
I20260929 19:03:14.033802 12952 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:03:14.034497 12952 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:03:14.043219 12952 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63bb191167ed46a992890d14681572cd: Generated new TSK 0
I20260929 19:03:14.043407 12952 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.158.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab.
Entry for principal kudu/127.8.158.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab.
I20260929 19:03:14.074277 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.65:0
--local_ip_for_outbound_sockets=127.8.158.65
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:43067
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab
--principal=kudu/127.8.158.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:32991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:14.157202 12964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:14.157405 12964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:14.157429 12964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:14.158601 12964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:14.158640 12964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:14.158667 12964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.65
I20260929 19:03:14.160145 12964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:32991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab
--principal=kudu/127.8.158.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:43067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.65
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:14.160373 12964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:14.160625 12964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:14.165490 12970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:14.165495 12969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:14.165524 12972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:14.165985 12964 server_base.cc:1034] running on GCE node
I20260929 19:03:14.166239 12964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:14.166518 12964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:14.167680 12964 hybrid_clock.cc:648] HybridClock initialized: now 1790708594167678 us; error 39 us; skew 500 ppm
Sep 29 19:03:14 dist-test-slave-nmsz krb5kdc[12858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708594, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:14.170291 12964 init.cc:377] Logged in from keytab as kudu/127.8.158.65@KRBTEST.COM (short username kudu)
I20260929 19:03:14.170689 12964 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:14.170766 12964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:14.170914 12964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:14.171990 12964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-0/data/instance:
uuid: "2679910f712d451fb3f5a027ade09b80"
format_stamp: "Formatted at 2026-09-29 19:03:14 on dist-test-slave-nmsz"
server_key: "dcd0a5204ae20595fb99d0ae79c63530"
server_key_iv: "a95945ebde5c64c23e7e90c82aa6906b"
server_key_version: "encryptionkey@0"
I20260929 19:03:14.172340 12964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance:
uuid: "2679910f712d451fb3f5a027ade09b80"
format_stamp: "Formatted at 2026-09-29 19:03:14 on dist-test-slave-nmsz"
server_key: "dcd0a5204ae20595fb99d0ae79c63530"
server_key_iv: "a95945ebde5c64c23e7e90c82aa6906b"
server_key_version: "encryptionkey@0"
I20260929 19:03:14.173602 12964 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:03:14.174309 12978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:14.174532 12964 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:14.174628 12964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-0/wal
uuid: "2679910f712d451fb3f5a027ade09b80"
format_stamp: "Formatted at 2026-09-29 19:03:14 on dist-test-slave-nmsz"
server_key: "dcd0a5204ae20595fb99d0ae79c63530"
server_key_iv: "a95945ebde5c64c23e7e90c82aa6906b"
server_key_version: "encryptionkey@0"
I20260929 19:03:14.174711 12964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:14.194643 12964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:14.196573 12964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:14.196738 12964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:14.196966 12964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:14.197273 12964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:14.197329 12964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:14.197372 12964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:14.197400 12964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:14.203210 12964 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.65:37117
I20260929 19:03:14.203251 13091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.65:37117 every 8 connection(s)
I20260929 19:03:14.203490 12964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
Sep 29 19:03:14 dist-test-slave-nmsz krb5kdc[12858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708594, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:03:14.209292 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 12964
I20260929 19:03:14.209407 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance
I20260929 19:03:14.209587 8825 external_mini_cluster.cc:1428] Setting key f6fa8f0a60c82fbfd1b3fa8453ec1f1a
I20260929 19:03:14.212558 12942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:14.204129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:51987 (local address 127.8.158.126:43067)
0929 19:03:14.204192 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:14.204194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:14.204393 (+ 199us) server_negotiation.cc:407] Connection header received
0929 19:03:14.204891 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:14.204892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:14.204905 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:14.204974 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:14.205873 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:14.206452 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:14.207387 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:14.207583 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:14.207643 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:14.209712 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:14.209714 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:14.209714 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:14.209727 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:14.211193 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:14.211595 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:14.211596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:14.211597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:14.211668 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:14.211911 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:14.211912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:14.211912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:14.212020 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:14.212094 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:14.212376 (+ 282us) server_negotiation.cc:299] Negotiation successful
0929 19:03:14.212468 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 19:03:14.212761 13094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:14.204100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:43067 (local address 127.8.158.65:51987)
0929 19:03:14.204261 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:14.204300 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:14.204770 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:14.205049 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:14.205052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:14.205303 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:14.205767 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:14.205771 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:14.206531 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:14.206532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:14.207307 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:14.207309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:14.207665 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:14.207666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:14.207763 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:14.208175 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:14.208183 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:14.209628 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:14.211291 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:14.211297 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:14.211299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:14.211506 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:14.211733 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:14.211734 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:14.211734 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:14.211844 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:14.212084 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:14.212089 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:14.212248 (+ 159us) client_negotiation.cc:769] Sending connection context
0929 19:03:14.212355 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 19:03:14.212521 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":77,"threads_started":1}
I20260929 19:03:14.213354 13092 heartbeater.cc:347] Connected to a master server at 127.8.158.126:43067
I20260929 19:03:14.213451 13092 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:14.213591 13092 heartbeater.cc:511] Master 127.8.158.126:43067 requested a full tablet report, sending...
I20260929 19:03:14.214118 12902 ts_manager.cc:194] Registered new tserver with Master: 2679910f712d451fb3f5a027ade09b80 (127.8.158.65:37117)
I20260929 19:03:14.215123 12902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.65@KRBTEST.COM'} at 127.8.158.65:51987
WARNING: no policy specified for kudu/127.8.158.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab.
Entry for principal kudu/127.8.158.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab.
I20260929 19:03:14.233981 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.66:0
--local_ip_for_outbound_sockets=127.8.158.66
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:43067
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab
--principal=kudu/127.8.158.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:32991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:14.313733 13097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:14.313972 13097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:14.314019 13097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:14.315198 13097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:14.315263 13097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:14.315310 13097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.66
I20260929 19:03:14.316784 13097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:32991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab
--principal=kudu/127.8.158.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:43067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.66
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:14.317040 13097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:14.317253 13097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:14.322237 13102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:14.322423 13103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:14.322243 13105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:14.322466 13097 server_base.cc:1034] running on GCE node
I20260929 19:03:14.322700 13097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:14.322942 13097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:14.324082 13097 hybrid_clock.cc:648] HybridClock initialized: now 1790708594324065 us; error 36 us; skew 500 ppm
Sep 29 19:03:14 dist-test-slave-nmsz krb5kdc[12858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708594, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:14.326725 13097 init.cc:377] Logged in from keytab as kudu/127.8.158.66@KRBTEST.COM (short username kudu)
I20260929 19:03:14.327055 13097 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:14.327127 13097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:14.327296 13097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:14.328358 13097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-1/data/instance:
uuid: "44aec0c8eec8431aa1b93023de9fa91d"
format_stamp: "Formatted at 2026-09-29 19:03:14 on dist-test-slave-nmsz"
server_key: "b610ee3cd255f1c6f7b758b750a88326"
server_key_iv: "43badf5d8cf3899fee974b447bef018a"
server_key_version: "encryptionkey@0"
I20260929 19:03:14.328709 13097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance:
uuid: "44aec0c8eec8431aa1b93023de9fa91d"
format_stamp: "Formatted at 2026-09-29 19:03:14 on dist-test-slave-nmsz"
server_key: "b610ee3cd255f1c6f7b758b750a88326"
server_key_iv: "43badf5d8cf3899fee974b447bef018a"
server_key_version: "encryptionkey@0"
I20260929 19:03:14.329980 13097 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 19:03:14.330785 13111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:14.330953 13097 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:14.331050 13097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-1/wal
uuid: "44aec0c8eec8431aa1b93023de9fa91d"
format_stamp: "Formatted at 2026-09-29 19:03:14 on dist-test-slave-nmsz"
server_key: "b610ee3cd255f1c6f7b758b750a88326"
server_key_iv: "43badf5d8cf3899fee974b447bef018a"
server_key_version: "encryptionkey@0"
I20260929 19:03:14.331125 13097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:14.355938 13097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:14.357806 13097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:14.357970 13097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:14.358198 13097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:14.358580 13097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:14.358639 13097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:14.358681 13097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:14.358700 13097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:14.364229 13097 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.66:33939
I20260929 19:03:14.364264 13224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.66:33939 every 8 connection(s)
I20260929 19:03:14.364542 13097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
I20260929 19:03:14.368710 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 13097
I20260929 19:03:14.368824 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance
I20260929 19:03:14.369029 8825 external_mini_cluster.cc:1428] Setting key 9c3ac416f87fdbecdd9d729d7a82a90c
Sep 29 19:03:14 dist-test-slave-nmsz krb5kdc[12858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708594, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.66@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:03:14.374291 12942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:14.365191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:43607 (local address 127.8.158.126:43067)
0929 19:03:14.365251 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:14.365254 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:14.365486 (+ 232us) server_negotiation.cc:407] Connection header received
0929 19:03:14.365955 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:14.365956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:14.365966 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:14.366014 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:14.366955 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:14.367450 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:14.368432 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:14.368665 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:14.368770 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:14.371365 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:14.371367 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:14.371367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:14.371376 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:14.372949 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:14.373333 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:14.373334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:14.373335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:14.373396 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:14.373618 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:14.373619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:14.373619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:14.373776 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:14.373855 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:14.374100 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 19:03:14.374182 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 19:03:14.374360 13227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:14.365191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:43067 (local address 127.8.158.66:43607)
0929 19:03:14.365347 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:14.365379 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:14.365873 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:14.366073 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:14.366077 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:14.366361 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:14.366849 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:14.366854 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:14.367570 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:14.367571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:14.368354 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:14.368356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:14.368780 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:14.368781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:14.368940 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:14.369400 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:14.369409 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:14.371265 (+ 1856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:14.373039 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:14.373044 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:14.373045 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:14.373252 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:14.373463 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:14.373464 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:14.373464 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:14.373552 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:14.373857 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:14.373861 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:14.373965 (+ 104us) client_negotiation.cc:769] Sending connection context
0929 19:03:14.374046 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 19:03:14.374163 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":77,"threads_started":1}
I20260929 19:03:14.374850 13225 heartbeater.cc:347] Connected to a master server at 127.8.158.126:43067
I20260929 19:03:14.374933 13225 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:14.375070 13225 heartbeater.cc:511] Master 127.8.158.126:43067 requested a full tablet report, sending...
I20260929 19:03:14.375425 12902 ts_manager.cc:194] Registered new tserver with Master: 44aec0c8eec8431aa1b93023de9fa91d (127.8.158.66:33939)
I20260929 19:03:14.375788 12902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.66@KRBTEST.COM'} at 127.8.158.66:43607
WARNING: no policy specified for kudu/127.8.158.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab.
Entry for principal kudu/127.8.158.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab.
I20260929 19:03:14.397388 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.67:0
--local_ip_for_outbound_sockets=127.8.158.67
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:43067
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab
--principal=kudu/127.8.158.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:32991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:14.476807 13230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:14.477023 13230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:14.477059 13230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:14.478247 13230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:14.478312 13230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:14.478384 13230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.67
I20260929 19:03:14.479848 13230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:32991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab
--principal=kudu/127.8.158.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:43067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.67
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:14.480100 13230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:14.480316 13230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:14.485113 13235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:14.485170 13236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:14.485283 13230 server_base.cc:1034] running on GCE node
W20260929 19:03:14.485112 13238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:14.485646 13230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:14.485884 13230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:14.487042 13230 hybrid_clock.cc:648] HybridClock initialized: now 1790708594487024 us; error 43 us; skew 500 ppm
Sep 29 19:03:14 dist-test-slave-nmsz krb5kdc[12858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708594, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:14.489718 13230 init.cc:377] Logged in from keytab as kudu/127.8.158.67@KRBTEST.COM (short username kudu)
I20260929 19:03:14.490115 13230 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:14.490180 13230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:14.490360 13230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:14.491461 13230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-2/data/instance:
uuid: "f8a46d0b10f7429da3e74f204d5a4198"
format_stamp: "Formatted at 2026-09-29 19:03:14 on dist-test-slave-nmsz"
server_key: "b351842947606c925ba044b2dfa10944"
server_key_iv: "a74f88e2befb92ffd8a544121a50cb94"
server_key_version: "encryptionkey@0"
I20260929 19:03:14.491909 13230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance:
uuid: "f8a46d0b10f7429da3e74f204d5a4198"
format_stamp: "Formatted at 2026-09-29 19:03:14 on dist-test-slave-nmsz"
server_key: "b351842947606c925ba044b2dfa10944"
server_key_iv: "a74f88e2befb92ffd8a544121a50cb94"
server_key_version: "encryptionkey@0"
I20260929 19:03:14.493275 13230 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:03:14.494110 13244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:14.494308 13230 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:14.494515 13230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-2/wal
uuid: "f8a46d0b10f7429da3e74f204d5a4198"
format_stamp: "Formatted at 2026-09-29 19:03:14 on dist-test-slave-nmsz"
server_key: "b351842947606c925ba044b2dfa10944"
server_key_iv: "a74f88e2befb92ffd8a544121a50cb94"
server_key_version: "encryptionkey@0"
I20260929 19:03:14.494609 13230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:14.518726 13230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:14.520669 13230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:14.520823 13230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:14.521060 13230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:14.521371 13230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:14.521405 13230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:14.521447 13230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:14.521474 13230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:14.527581 13230 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.67:38111
I20260929 19:03:14.527647 13357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.67:38111 every 8 connection(s)
I20260929 19:03:14.527886 13230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
I20260929 19:03:14.532269 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 13230
I20260929 19:03:14.532359 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance
I20260929 19:03:14.532609 8825 external_mini_cluster.cc:1428] Setting key 997bae036d4a46b8718a6e98f58b236e
Sep 29 19:03:14 dist-test-slave-nmsz krb5kdc[12858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708594, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.67@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:03:14.537133 12942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:14.528571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:54127 (local address 127.8.158.126:43067)
0929 19:03:14.528642 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:14.528645 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:14.528790 (+ 145us) server_negotiation.cc:407] Connection header received
0929 19:03:14.529283 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:14.529284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:14.529293 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:14.529340 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:14.530391 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:14.530890 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:14.531938 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:14.532130 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:14.532185 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:14.534621 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:14.534623 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:14.534623 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:14.534629 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:14.535756 (+ 1127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:14.536145 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:14.536146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:14.536147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:14.536208 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:14.536515 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:14.536516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:14.536516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:14.536685 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:14.536753 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:14.536999 (+ 246us) server_negotiation.cc:299] Negotiation successful
0929 19:03:14.537068 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 19:03:14.537313 13360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:14.528517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:43067 (local address 127.8.158.67:54127)
0929 19:03:14.528688 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:14.528724 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:14.529200 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:14.529401 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:14.529405 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:14.529732 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:14.530248 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:14.530254 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:14.530974 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:14.530975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:14.531862 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:14.531864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:14.532221 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:14.532223 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:14.532379 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:14.532923 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:14.532932 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:14.534559 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:14.535825 (+ 1266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:14.535830 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:14.535832 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:14.536033 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:14.536299 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:14.536300 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:14.536300 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:14.536413 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:14.536741 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:14.536744 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:14.536868 (+ 124us) client_negotiation.cc:769] Sending connection context
0929 19:03:14.536956 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 19:03:14.537079 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":88,"threads_started":1}
I20260929 19:03:14.537915 13358 heartbeater.cc:347] Connected to a master server at 127.8.158.126:43067
I20260929 19:03:14.538029 13358 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:14.538179 13358 heartbeater.cc:511] Master 127.8.158.126:43067 requested a full tablet report, sending...
I20260929 19:03:14.538604 12902 ts_manager.cc:194] Registered new tserver with Master: f8a46d0b10f7429da3e74f204d5a4198 (127.8.158.67:38111)
I20260929 19:03:14.539006 12902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.67@KRBTEST.COM'} at 127.8.158.67:54127
I20260929 19:03:14.544382 8825 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:03:14.551308 12942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:14.545617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47514 (local address 127.8.158.126:43067)
0929 19:03:14.545650 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:14.545652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:14.545770 (+ 118us) server_negotiation.cc:407] Connection header received
0929 19:03:14.545821 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:14.545822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:14.545829 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:14.545877 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:14.546455 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:14.546912 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:14.547623 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:14.547838 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:14.547902 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:14.549010 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:14.549011 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:14.549012 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:14.549018 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:14.550256 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:14.550648 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:14.550649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:14.550650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:14.550725 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:14.550908 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:14.550909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:14.550909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:14.551032 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:14.551107 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:14.551196 (+ 89us) server_negotiation.cc:299] Negotiation successful
0929 19:03:14.551253 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260929 19:03:14.552802 12902 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47514:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:03:14.554379 12902 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:03:14.560536 13370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:14.556743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.66:33939 (local address 127.0.0.1:51340)
0929 19:03:14.556910 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:14.556943 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:14.557172 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:14.557400 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:14.557401 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:14.557403 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:14.557445 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:14.557447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:14.558670 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:14.558670 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:14.559473 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:14.559476 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:14.560254 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:14.560255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:14.560330 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:14.560343 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 19:03:14.560416 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 19:03:14.560464 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":92,"threads_started":1}
I20260929 19:03:14.560547 13374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:14.556983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51340 (local address 127.8.158.66:33939)
0929 19:03:14.557153 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:14.557157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:14.557167 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:03:14.557232 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:14.557234 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:14.557249 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:14.557316 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:14.557522 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:14.558586 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:14.559568 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:14.560166 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:14.560255 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:14.560312 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:14.560429 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 19:03:14.560465 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":83,"threads_started":1}
I20260929 19:03:14.561043 13373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:14.556542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.67:38111 (local address 127.0.0.1:45050)
0929 19:03:14.556889 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:14.556900 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:14.556951 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:14.557292 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:14.557293 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:14.557295 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:14.557342 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:14.557344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:14.558630 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:14.558631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:14.559824 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:14.559826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:14.560808 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:14.560809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:14.560871 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:14.560875 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:03:14.560933 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 19:03:14.560970 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":5,"thread_start_us":93,"threads_started":1}
I20260929 19:03:14.561087 13372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:14.556672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45050 (local address 127.8.158.67:38111)
0929 19:03:14.557140 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:14.557144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:14.557153 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:03:14.557188 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:14.557189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:14.557204 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:14.557284 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:14.557406 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:14.558511 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:14.559944 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:14.560712 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:14.560809 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:14.560870 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:14.560946 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 19:03:14.560987 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":69,"threads_started":1}
I20260929 19:03:14.561177 13369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:14.556504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:37117 (local address 127.0.0.1:36104)
0929 19:03:14.556674 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:14.556714 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:14.556780 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:14.557013 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:14.557016 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:14.557019 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:14.557086 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:14.557088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:14.558384 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:14.558385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:14.559824 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:14.559826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:14.560929 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:14.560930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:14.561008 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:14.561013 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 19:03:14.561048 (+ 35us) client_negotiation.cc:240] Negotiation successful
0929 19:03:14.561089 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":44,"threads_started":1}
I20260929 19:03:14.561364 13159 tablet_service.cc:1467] Processing CreateTablet for tablet 235eab1887ea4520a2c0c216ac280d84 (DEFAULT_TABLE table=test-table [id=5da2f14c4c6a4bc19a48d8543fb9302d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:14.561578 13371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:14.556606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36104 (local address 127.8.158.65:37117)
0929 19:03:14.556802 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:14.556808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:14.556824 (+ 16us) server_negotiation.cc:407] Connection header received
0929 19:03:14.556883 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:14.556885 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:14.556902 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:14.556987 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:14.557266 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:14.558243 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:14.559938 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:14.560712 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:14.561361 (+ 649us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:14.561439 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:14.561467 (+ 28us) server_negotiation.cc:299] Negotiation successful
0929 19:03:14.561506 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":92,"threads_started":1}
I20260929 19:03:14.561702 13159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 235eab1887ea4520a2c0c216ac280d84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:14.561839 13292 tablet_service.cc:1467] Processing CreateTablet for tablet 235eab1887ea4520a2c0c216ac280d84 (DEFAULT_TABLE table=test-table [id=5da2f14c4c6a4bc19a48d8543fb9302d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:14.562224 13292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 235eab1887ea4520a2c0c216ac280d84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:14.562904 13026 tablet_service.cc:1467] Processing CreateTablet for tablet 235eab1887ea4520a2c0c216ac280d84 (DEFAULT_TABLE table=test-table [id=5da2f14c4c6a4bc19a48d8543fb9302d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:14.563194 13026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 235eab1887ea4520a2c0c216ac280d84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:14.565339 13375 tablet_bootstrap.cc:492] T 235eab1887ea4520a2c0c216ac280d84 P 44aec0c8eec8431aa1b93023de9fa91d: Bootstrap starting.
I20260929 19:03:14.565503 13376 tablet_bootstrap.cc:492] T 235eab1887ea4520a2c0c216ac280d84 P f8a46d0b10f7429da3e74f204d5a4198: Bootstrap starting.
I20260929 19:03:14.565997 13377 tablet_bootstrap.cc:492] T 235eab1887ea4520a2c0c216ac280d84 P 2679910f712d451fb3f5a027ade09b80: Bootstrap starting.
I20260929 19:03:14.566056 13375 tablet_bootstrap.cc:654] T 235eab1887ea4520a2c0c216ac280d84 P 44aec0c8eec8431aa1b93023de9fa91d: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:14.566133 13376 tablet_bootstrap.cc:654] T 235eab1887ea4520a2c0c216ac280d84 P f8a46d0b10f7429da3e74f204d5a4198: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:14.566689 13375 log.cc:824] T 235eab1887ea4520a2c0c216ac280d84 P 44aec0c8eec8431aa1b93023de9fa91d: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:14.566774 13377 tablet_bootstrap.cc:654] T 235eab1887ea4520a2c0c216ac280d84 P 2679910f712d451fb3f5a027ade09b80: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:14.566824 13376 log.cc:824] T 235eab1887ea4520a2c0c216ac280d84 P f8a46d0b10f7429da3e74f204d5a4198: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:14.567386 13377 log.cc:824] T 235eab1887ea4520a2c0c216ac280d84 P 2679910f712d451fb3f5a027ade09b80: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:14.567505 13375 tablet_bootstrap.cc:492] T 235eab1887ea4520a2c0c216ac280d84 P 44aec0c8eec8431aa1b93023de9fa91d: No bootstrap required, opened a new log
I20260929 19:03:14.567580 13375 ts_tablet_manager.cc:1397] T 235eab1887ea4520a2c0c216ac280d84 P 44aec0c8eec8431aa1b93023de9fa91d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:03:14.567610 13376 tablet_bootstrap.cc:492] T 235eab1887ea4520a2c0c216ac280d84 P f8a46d0b10f7429da3e74f204d5a4198: No bootstrap required, opened a new log
I20260929 19:03:14.567672 13376 ts_tablet_manager.cc:1397] T 235eab1887ea4520a2c0c216ac280d84 P f8a46d0b10f7429da3e74f204d5a4198: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:03:14.568084 13377 tablet_bootstrap.cc:492] T 235eab1887ea4520a2c0c216ac280d84 P 2679910f712d451fb3f5a027ade09b80: No bootstrap required, opened a new log
I20260929 19:03:14.568192 13377 ts_tablet_manager.cc:1397] T 235eab1887ea4520a2c0c216ac280d84 P 2679910f712d451fb3f5a027ade09b80: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260929 19:03:14.569195 13376 raft_consensus.cc:348] T 235eab1887ea4520a2c0c216ac280d84 P f8a46d0b10f7429da3e74f204d5a4198 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8a46d0b10f7429da3e74f204d5a4198" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38111 } } peers { permanent_uuid: "44aec0c8eec8431aa1b93023de9fa91d" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 33939 } } peers { permanent_uuid: "2679910f712d451fb3f5a027ade09b80" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 37117 } }
I20260929 19:03:14.569345 13376 raft_consensus.cc:374] T 235eab1887ea4520a2c0c216ac280d84 P f8a46d0b10f7429da3e74f204d5a4198 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:14.569406 13376 raft_consensus.cc:729] T 235eab1887ea4520a2c0c216ac280d84 P f8a46d0b10f7429da3e74f204d5a4198 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8a46d0b10f7429da3e74f204d5a4198, State: Initialized, Role: FOLLOWER
I20260929 19:03:14.569525 13376 consensus_queue.cc:260] T 235eab1887ea4520a2c0c216ac280d84 P f8a46d0b10f7429da3e74f204d5a4198 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8a46d0b10f7429da3e74f204d5a4198" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38111 } } peers { permanent_uuid: "44aec0c8eec8431aa1b93023de9fa91d" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 33939 } } peers { permanent_uuid: "2679910f712d451fb3f5a027ade09b80" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 37117 } }
I20260929 19:03:14.569577 13375 raft_consensus.cc:348] T 235eab1887ea4520a2c0c216ac280d84 P 44aec0c8eec8431aa1b93023de9fa91d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8a46d0b10f7429da3e74f204d5a4198" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38111 } } peers { permanent_uuid: "44aec0c8eec8431aa1b93023de9fa91d" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 33939 } } peers { permanent_uuid: "2679910f712d451fb3f5a027ade09b80" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 37117 } }
I20260929 19:03:14.569758 13375 raft_consensus.cc:374] T 235eab1887ea4520a2c0c216ac280d84 P 44aec0c8eec8431aa1b93023de9fa91d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:14.569804 13375 raft_consensus.cc:729] T 235eab1887ea4520a2c0c216ac280d84 P 44aec0c8eec8431aa1b93023de9fa91d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44aec0c8eec8431aa1b93023de9fa91d, State: Initialized, Role: FOLLOWER
I20260929 19:03:14.569815 13376 ts_tablet_manager.cc:1428] T 235eab1887ea4520a2c0c216ac280d84 P f8a46d0b10f7429da3e74f204d5a4198: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:03:14.569939 13358 heartbeater.cc:503] Master 127.8.158.126:43067 was elected leader, sending a full tablet report...
I20260929 19:03:14.569918 13375 consensus_queue.cc:260] T 235eab1887ea4520a2c0c216ac280d84 P 44aec0c8eec8431aa1b93023de9fa91d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8a46d0b10f7429da3e74f204d5a4198" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38111 } } peers { permanent_uuid: "44aec0c8eec8431aa1b93023de9fa91d" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 33939 } } peers { permanent_uuid: "2679910f712d451fb3f5a027ade09b80" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 37117 } }
I20260929 19:03:14.570173 13225 heartbeater.cc:503] Master 127.8.158.126:43067 was elected leader, sending a full tablet report...
I20260929 19:03:14.570182 13375 ts_tablet_manager.cc:1428] T 235eab1887ea4520a2c0c216ac280d84 P 44aec0c8eec8431aa1b93023de9fa91d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 19:03:14.570178 13377 raft_consensus.cc:348] T 235eab1887ea4520a2c0c216ac280d84 P 2679910f712d451fb3f5a027ade09b80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8a46d0b10f7429da3e74f204d5a4198" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38111 } } peers { permanent_uuid: "44aec0c8eec8431aa1b93023de9fa91d" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 33939 } } peers { permanent_uuid: "2679910f712d451fb3f5a027ade09b80" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 37117 } }
I20260929 19:03:14.570370 13377 raft_consensus.cc:374] T 235eab1887ea4520a2c0c216ac280d84 P 2679910f712d451fb3f5a027ade09b80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:14.570405 13377 raft_consensus.cc:729] T 235eab1887ea4520a2c0c216ac280d84 P 2679910f712d451fb3f5a027ade09b80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2679910f712d451fb3f5a027ade09b80, State: Initialized, Role: FOLLOWER
I20260929 19:03:14.570607 13377 consensus_queue.cc:260] T 235eab1887ea4520a2c0c216ac280d84 P 2679910f712d451fb3f5a027ade09b80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8a46d0b10f7429da3e74f204d5a4198" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38111 } } peers { permanent_uuid: "44aec0c8eec8431aa1b93023de9fa91d" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 33939 } } peers { permanent_uuid: "2679910f712d451fb3f5a027ade09b80" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 37117 } }
I20260929 19:03:14.570956 13377 ts_tablet_manager.cc:1428] T 235eab1887ea4520a2c0c216ac280d84 P 2679910f712d451fb3f5a027ade09b80: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 19:03:14.571067 13092 heartbeater.cc:503] Master 127.8.158.126:43067 was elected leader, sending a full tablet report...
W20260929 19:03:14.615056 13226 tablet.cc:2369] T 235eab1887ea4520a2c0c216ac280d84 P 44aec0c8eec8431aa1b93023de9fa91d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 19:03:14.704214 13093 tablet.cc:2369] T 235eab1887ea4520a2c0c216ac280d84 P 2679910f712d451fb3f5a027ade09b80: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:03:14.707856 13381 raft_consensus.cc:482] T 235eab1887ea4520a2c0c216ac280d84 P f8a46d0b10f7429da3e74f204d5a4198 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:03:14.707968 13381 raft_consensus.cc:504] T 235eab1887ea4520a2c0c216ac280d84 P f8a46d0b10f7429da3e74f204d5a4198 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8a46d0b10f7429da3e74f204d5a4198" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38111 } } peers { permanent_uuid: "44aec0c8eec8431aa1b93023de9fa91d" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 33939 } } peers { permanent_uuid: "2679910f712d451fb3f5a027ade09b80" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 37117 } }
I20260929 19:03:14.708290 13381 leader_election.cc:288] T 235eab1887ea4520a2c0c216ac280d84 P f8a46d0b10f7429da3e74f204d5a4198 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44aec0c8eec8431aa1b93023de9fa91d (127.8.158.66:33939), 2679910f712d451fb3f5a027ade09b80 (127.8.158.65:37117)
I20260929 19:03:14.711675 13371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:14.708707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:32907 (local address 127.8.158.65:37117)
0929 19:03:14.708748 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:14.708749 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:14.708759 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:03:14.708814 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:14.708815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:14.708825 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:14.708889 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:14.709095 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:14.709614 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:14.710797 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:14.711320 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:14.711374 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:14.711381 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:14.711593 (+ 212us) server_negotiation.cc:299] Negotiation successful
0929 19:03:14.711628 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 19:03:14.711670 13360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:14.708603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:37117 (local address 127.8.158.67:32907)
0929 19:03:14.708672 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:14.708687 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:14.708745 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:14.708927 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:14.708928 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:14.708931 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:14.709005 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:14.709007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:14.709705 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:14.709706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:14.710697 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:14.710699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:14.711404 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:14.711405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:14.711486 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:14.711491 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 19:03:14.711561 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 19:03:14.711597 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 19:03:14.712031 13046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "235eab1887ea4520a2c0c216ac280d84" candidate_uuid: "f8a46d0b10f7429da3e74f204d5a4198" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2679910f712d451fb3f5a027ade09b80" is_pre_election: true
I20260929 19:03:14.712172 13384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:14.708615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.66:33939 (local address 127.8.158.67:40739)
0929 19:03:14.708865 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:14.708874 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:14.708923 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:14.709095 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:14.709095 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:14.709097 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:14.709164 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:14.709165 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:14.710167 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:14.710168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:14.711126 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:14.711129 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:14.711920 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:14.711922 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:14.711993 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:14.711997 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:03:14.712054 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 19:03:14.712094 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":79,"thread_start_us":95,"threads_started":1}
I20260929 19:03:14.712167 13374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:14.708740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:40739 (local address 127.8.158.66:33939)
0929 19:03:14.708787 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:14.708789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:14.708982 (+ 193us) server_negotiation.cc:407] Connection header received
0929 19:03:14.709010 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:14.709011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:14.709021 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:14.709073 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:14.709392 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:14.710064 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:14.711234 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:14.711812 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:14.711873 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:14.711881 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:14.712073 (+ 192us) server_negotiation.cc:299] Negotiation successful
0929 19:03:14.712108 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 19:03:14.712204 13046 raft_consensus.cc:2454] T 235eab1887ea4520a2c0c216ac280d84 P 2679910f712d451fb3f5a027ade09b80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8a46d0b10f7429da3e74f204d5a4198 in term 0.
I20260929 19:03:14.712442 13245 leader_election.cc:302] T 235eab1887ea4520a2c0c216ac280d84 P f8a46d0b10f7429da3e74f204d5a4198 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2679910f712d451fb3f5a027ade09b80, f8a46d0b10f7429da3e74f204d5a4198; no voters:
I20260929 19:03:14.712585 13381 raft_consensus.cc:2781] T 235eab1887ea4520a2c0c216ac280d84 P f8a46d0b10f7429da3e74f204d5a4198 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:03:14.712653 13381 raft_consensus.cc:482] T 235eab1887ea4520a2c0c216ac280d84 P f8a46d0b10f7429da3e74f204d5a4198 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:03:14.712684 13381 raft_consensus.cc:3037] T 235eab1887ea4520a2c0c216ac280d84 P f8a46d0b10f7429da3e74f204d5a4198 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:14.712741 13179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "235eab1887ea4520a2c0c216ac280d84" candidate_uuid: "f8a46d0b10f7429da3e74f204d5a4198" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44aec0c8eec8431aa1b93023de9fa91d" is_pre_election: true
I20260929 19:03:14.712905 13179 raft_consensus.cc:2454] T 235eab1887ea4520a2c0c216ac280d84 P 44aec0c8eec8431aa1b93023de9fa91d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8a46d0b10f7429da3e74f204d5a4198 in term 0.
I20260929 19:03:14.713580 13381 raft_consensus.cc:504] T 235eab1887ea4520a2c0c216ac280d84 P f8a46d0b10f7429da3e74f204d5a4198 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8a46d0b10f7429da3e74f204d5a4198" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38111 } } peers { permanent_uuid: "44aec0c8eec8431aa1b93023de9fa91d" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 33939 } } peers { permanent_uuid: "2679910f712d451fb3f5a027ade09b80" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 37117 } }
I20260929 19:03:14.713732 13381 leader_election.cc:288] T 235eab1887ea4520a2c0c216ac280d84 P f8a46d0b10f7429da3e74f204d5a4198 [CANDIDATE]: Term 1 election: Requested vote from peers 44aec0c8eec8431aa1b93023de9fa91d (127.8.158.66:33939), 2679910f712d451fb3f5a027ade09b80 (127.8.158.65:37117)
I20260929 19:03:14.713878 13046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "235eab1887ea4520a2c0c216ac280d84" candidate_uuid: "f8a46d0b10f7429da3e74f204d5a4198" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2679910f712d451fb3f5a027ade09b80"
I20260929 19:03:14.713897 13179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "235eab1887ea4520a2c0c216ac280d84" candidate_uuid: "f8a46d0b10f7429da3e74f204d5a4198" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44aec0c8eec8431aa1b93023de9fa91d"
I20260929 19:03:14.713959 13046 raft_consensus.cc:3037] T 235eab1887ea4520a2c0c216ac280d84 P 2679910f712d451fb3f5a027ade09b80 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:14.713982 13179 raft_consensus.cc:3037] T 235eab1887ea4520a2c0c216ac280d84 P 44aec0c8eec8431aa1b93023de9fa91d [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:14.714726 13046 raft_consensus.cc:2454] T 235eab1887ea4520a2c0c216ac280d84 P 2679910f712d451fb3f5a027ade09b80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8a46d0b10f7429da3e74f204d5a4198 in term 1.
I20260929 19:03:14.714728 13179 raft_consensus.cc:2454] T 235eab1887ea4520a2c0c216ac280d84 P 44aec0c8eec8431aa1b93023de9fa91d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8a46d0b10f7429da3e74f204d5a4198 in term 1.
I20260929 19:03:14.714898 13247 leader_election.cc:302] T 235eab1887ea4520a2c0c216ac280d84 P f8a46d0b10f7429da3e74f204d5a4198 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44aec0c8eec8431aa1b93023de9fa91d, f8a46d0b10f7429da3e74f204d5a4198; no voters:
I20260929 19:03:14.715019 13381 raft_consensus.cc:2781] T 235eab1887ea4520a2c0c216ac280d84 P f8a46d0b10f7429da3e74f204d5a4198 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:03:14.715178 13381 raft_consensus.cc:686] T 235eab1887ea4520a2c0c216ac280d84 P f8a46d0b10f7429da3e74f204d5a4198 [term 1 LEADER]: Becoming Leader. State: Replica: f8a46d0b10f7429da3e74f204d5a4198, State: Running, Role: LEADER
I20260929 19:03:14.715279 13381 consensus_queue.cc:237] T 235eab1887ea4520a2c0c216ac280d84 P f8a46d0b10f7429da3e74f204d5a4198 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8a46d0b10f7429da3e74f204d5a4198" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38111 } } peers { permanent_uuid: "44aec0c8eec8431aa1b93023de9fa91d" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 33939 } } peers { permanent_uuid: "2679910f712d451fb3f5a027ade09b80" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 37117 } }
I20260929 19:03:14.716038 12902 catalog_manager.cc:5495] T 235eab1887ea4520a2c0c216ac280d84 P f8a46d0b10f7429da3e74f204d5a4198 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8a46d0b10f7429da3e74f204d5a4198 (127.8.158.67). New cstate: current_term: 1 leader_uuid: "f8a46d0b10f7429da3e74f204d5a4198" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8a46d0b10f7429da3e74f204d5a4198" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44aec0c8eec8431aa1b93023de9fa91d" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 33939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2679910f712d451fb3f5a027ade09b80" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 37117 } health_report { overall_health: UNKNOWN } } }
I20260929 19:03:14.741679 13372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:14.739228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45062 (local address 127.8.158.67:38111)
0929 19:03:14.739293 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:14.739299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:14.739307 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:03:14.739339 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:14.739339 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:14.739355 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:14.739422 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:03:14.739751 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:14.740217 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:14.740989 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:14.741175 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:14.741235 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:14.741381 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:03:14.741479 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:03:14.741566 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:14.741608 (+ 42us) server_negotiation.cc:299] Negotiation successful
0929 19:03:14.741624 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 19:03:14.743589 13046 raft_consensus.cc:1261] T 235eab1887ea4520a2c0c216ac280d84 P 2679910f712d451fb3f5a027ade09b80 [term 1 FOLLOWER]: Refusing update from remote peer f8a46d0b10f7429da3e74f204d5a4198: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:03:14.743621 13179 raft_consensus.cc:1261] T 235eab1887ea4520a2c0c216ac280d84 P 44aec0c8eec8431aa1b93023de9fa91d [term 1 FOLLOWER]: Refusing update from remote peer f8a46d0b10f7429da3e74f204d5a4198: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:03:14.743970 13381 consensus_queue.cc:1035] T 235eab1887ea4520a2c0c216ac280d84 P f8a46d0b10f7429da3e74f204d5a4198 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44aec0c8eec8431aa1b93023de9fa91d" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 33939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:03:14.744103 13385 consensus_queue.cc:1035] T 235eab1887ea4520a2c0c216ac280d84 P f8a46d0b10f7429da3e74f204d5a4198 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2679910f712d451fb3f5a027ade09b80" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 37117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:03:14.747340 13389 mvcc.cc:205] Tried to move back new op lower bound from 7334742404066988032 to 7334742403954438144. Current Snapshot: MvccSnapshot[applied={T|T < 7334742404066988032 or (T in {7334742404066988032})}]
I20260929 19:03:14.748571 12902 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:03:14.748665 12902 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:03:14.749454 12902 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 63bb191167ed46a992890d14681572cd: Sending DeleteTablet for 3 replicas of tablet 235eab1887ea4520a2c0c216ac280d84
I20260929 19:03:14.749732 13292 tablet_service.cc:1514] Processing DeleteTablet for tablet 235eab1887ea4520a2c0c216ac280d84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:14 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:45050
I20260929 19:03:14.749752 13026 tablet_service.cc:1514] Processing DeleteTablet for tablet 235eab1887ea4520a2c0c216ac280d84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:14 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:36104
I20260929 19:03:14.749781 13159 tablet_service.cc:1514] Processing DeleteTablet for tablet 235eab1887ea4520a2c0c216ac280d84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:14 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:51340
I20260929 19:03:14.749967 13397 tablet_replica.cc:330] T 235eab1887ea4520a2c0c216ac280d84 P f8a46d0b10f7429da3e74f204d5a4198: stopping tablet replica
I20260929 19:03:14.750067 13397 raft_consensus.cc:2229] T 235eab1887ea4520a2c0c216ac280d84 P f8a46d0b10f7429da3e74f204d5a4198 [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:03:14.750156 13397 raft_consensus.cc:2258] T 235eab1887ea4520a2c0c216ac280d84 P f8a46d0b10f7429da3e74f204d5a4198 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:03:14.750272 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 12964
I20260929 19:03:14.750299 13399 tablet_replica.cc:330] T 235eab1887ea4520a2c0c216ac280d84 P 44aec0c8eec8431aa1b93023de9fa91d: stopping tablet replica
I20260929 19:03:14.750412 13399 raft_consensus.cc:2229] T 235eab1887ea4520a2c0c216ac280d84 P 44aec0c8eec8431aa1b93023de9fa91d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:03:14.750509 13399 raft_consensus.cc:2258] T 235eab1887ea4520a2c0c216ac280d84 P 44aec0c8eec8431aa1b93023de9fa91d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:03:14.751163 13397 ts_tablet_manager.cc:1905] T 235eab1887ea4520a2c0c216ac280d84 P f8a46d0b10f7429da3e74f204d5a4198: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:03:14.751277 13399 ts_tablet_manager.cc:1905] T 235eab1887ea4520a2c0c216ac280d84 P 44aec0c8eec8431aa1b93023de9fa91d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:03:14.752827 13399 ts_tablet_manager.cc:1918] T 235eab1887ea4520a2c0c216ac280d84 P 44aec0c8eec8431aa1b93023de9fa91d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:03:14.752885 13399 log.cc:1196] T 235eab1887ea4520a2c0c216ac280d84 P 44aec0c8eec8431aa1b93023de9fa91d: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-1/wal/wals/235eab1887ea4520a2c0c216ac280d84
I20260929 19:03:14.753072 13397 ts_tablet_manager.cc:1918] T 235eab1887ea4520a2c0c216ac280d84 P f8a46d0b10f7429da3e74f204d5a4198: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:03:14.753131 13397 log.cc:1196] T 235eab1887ea4520a2c0c216ac280d84 P f8a46d0b10f7429da3e74f204d5a4198: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708573736807-8825-0/minicluster-data/ts-2/wal/wals/235eab1887ea4520a2c0c216ac280d84
I20260929 19:03:14.753152 13399 ts_tablet_manager.cc:1939] T 235eab1887ea4520a2c0c216ac280d84 P 44aec0c8eec8431aa1b93023de9fa91d: Deleting consensus metadata
I20260929 19:03:14.753386 13397 ts_tablet_manager.cc:1939] T 235eab1887ea4520a2c0c216ac280d84 P f8a46d0b10f7429da3e74f204d5a4198: Deleting consensus metadata
I20260929 19:03:14.753625 12888 catalog_manager.cc:4841] TS 44aec0c8eec8431aa1b93023de9fa91d (127.8.158.66:33939): tablet 235eab1887ea4520a2c0c216ac280d84 (table test-table [id=5da2f14c4c6a4bc19a48d8543fb9302d]) successfully deleted
I20260929 19:03:14.753826 12888 catalog_manager.cc:4841] TS f8a46d0b10f7429da3e74f204d5a4198 (127.8.158.67:38111): tablet 235eab1887ea4520a2c0c216ac280d84 (table test-table [id=5da2f14c4c6a4bc19a48d8543fb9302d]) successfully deleted
W20260929 19:03:14.756901 12886 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.158.65:37117 (error 108)
I20260929 19:03:14.757395 13369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:14.757264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:37117 (local address 127.0.0.1:36108)
0929 19:03:14.757324 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:14.757363 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.65:37117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260929 19:03:14.757422 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 13097
W20260929 19:03:14.759191 12886 catalog_manager.cc:4584] TS 2679910f712d451fb3f5a027ade09b80 (127.8.158.65:37117): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 235eab1887ea4520a2c0c216ac280d84: Network error: Client connection negotiation failed: client connection to 127.8.158.65:37117: connect: Connection refused (error 111)
I20260929 19:03:14.762555 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 13230
I20260929 19:03:14.767011 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 12871
2026-09-29T19:03:14Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2049 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T19:03:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:03:14Z Disabled control of system clock
I20260929 19:03:14.793177 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:37869
--webserver_interface=127.8.158.126
--webserver_port=0
--builtin_ntp_servers=127.8.158.84:40419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:37869
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 19:03:14.872414 13404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:14.872618 13404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:14.872639 13404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:14.873677 13404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:03:14.873710 13404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:14.873723 13404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:14.873735 13404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:03:14.873747 13404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:03:14.875048 13404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:40419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:37869
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:37869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.158.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:14.875227 13404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:14.875449 13404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:14.880097 13409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:14.880168 13410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:14.880096 13412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:14.880440 13404 server_base.cc:1034] running on GCE node
I20260929 19:03:14.880748 13404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:14.881003 13404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:14.882150 13404 hybrid_clock.cc:648] HybridClock initialized: now 1790708594882136 us; error 28 us; skew 500 ppm
I20260929 19:03:14.883378 13404 webserver.cc:460] Webserver started at http://127.8.158.126:36783/ using document root <none> and password file <none>
I20260929 19:03:14.883605 13404 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:14.883673 13404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:14.883842 13404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:14.884860 13404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "9fb63ccf9e394100a3da0ba8aad34f19"
format_stamp: "Formatted at 2026-09-29 19:03:14 on dist-test-slave-nmsz"
server_key: "3c7ed59e8e17a1d5a186e7d45ba9b3f5"
server_key_iv: "3bb9bb0f006b60ccb4d15ee4ad910473"
server_key_version: "encryptionkey@0"
I20260929 19:03:14.885205 13404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "9fb63ccf9e394100a3da0ba8aad34f19"
format_stamp: "Formatted at 2026-09-29 19:03:14 on dist-test-slave-nmsz"
server_key: "3c7ed59e8e17a1d5a186e7d45ba9b3f5"
server_key_iv: "3bb9bb0f006b60ccb4d15ee4ad910473"
server_key_version: "encryptionkey@0"
I20260929 19:03:14.886703 13404 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:03:14.887446 13418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:14.887647 13404 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:03:14.887714 13404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "9fb63ccf9e394100a3da0ba8aad34f19"
format_stamp: "Formatted at 2026-09-29 19:03:14 on dist-test-slave-nmsz"
server_key: "3c7ed59e8e17a1d5a186e7d45ba9b3f5"
server_key_iv: "3bb9bb0f006b60ccb4d15ee4ad910473"
server_key_version: "encryptionkey@0"
I20260929 19:03:14.887799 13404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:14.891027 13404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:14.891258 13404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:14.891400 13404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:14.895028 13404 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.126:37869
I20260929 19:03:14.895058 13470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.126:37869 every 8 connection(s)
I20260929 19:03:14.895354 13404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:03:14.895982 13471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:14.897133 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 13404
I20260929 19:03:14.897235 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:03:14.897403 8825 external_mini_cluster.cc:1428] Setting key 1654ffb4a43d8bff8baccdfe718399df
I20260929 19:03:14.899550 13471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fb63ccf9e394100a3da0ba8aad34f19: Bootstrap starting.
I20260929 19:03:14.900029 13474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:14.897912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56640 (local address 127.8.158.126:37869)
0929 19:03:14.898131 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:14.898139 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:14.898168 (+ 29us) server_negotiation.cc:407] Connection header received
0929 19:03:14.898726 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:14.898734 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:14.898786 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:14.898927 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:14.899554 (+ 627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:14.899558 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:14.899558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:03:14.899562 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:14.899606 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:03:14.899619 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:14.899727 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:14.899780 (+ 53us) server_negotiation.cc:299] Negotiation successful
0929 19:03:14.899834 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":55,"threads_started":1}
I20260929 19:03:14.900280 13471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fb63ccf9e394100a3da0ba8aad34f19: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:14.900576 13471 log.cc:824] T 00000000000000000000000000000000 P 9fb63ccf9e394100a3da0ba8aad34f19: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:14.901224 13471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fb63ccf9e394100a3da0ba8aad34f19: No bootstrap required, opened a new log
I20260929 19:03:14.902581 13471 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fb63ccf9e394100a3da0ba8aad34f19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fb63ccf9e394100a3da0ba8aad34f19" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37869 } }
I20260929 19:03:14.902715 13471 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fb63ccf9e394100a3da0ba8aad34f19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:14.902737 13471 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fb63ccf9e394100a3da0ba8aad34f19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fb63ccf9e394100a3da0ba8aad34f19, State: Initialized, Role: FOLLOWER
I20260929 19:03:14.902879 13471 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fb63ccf9e394100a3da0ba8aad34f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fb63ccf9e394100a3da0ba8aad34f19" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37869 } }
I20260929 19:03:14.902946 13471 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9fb63ccf9e394100a3da0ba8aad34f19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:03:14.902966 13471 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fb63ccf9e394100a3da0ba8aad34f19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:03:14.902992 13471 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fb63ccf9e394100a3da0ba8aad34f19 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:14.903538 13471 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fb63ccf9e394100a3da0ba8aad34f19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fb63ccf9e394100a3da0ba8aad34f19" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37869 } }
I20260929 19:03:14.903643 13471 leader_election.cc:302] T 00000000000000000000000000000000 P 9fb63ccf9e394100a3da0ba8aad34f19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fb63ccf9e394100a3da0ba8aad34f19; no voters:
I20260929 19:03:14.903811 13471 leader_election.cc:288] T 00000000000000000000000000000000 P 9fb63ccf9e394100a3da0ba8aad34f19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:03:14.904053 13476 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fb63ccf9e394100a3da0ba8aad34f19 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:03:14.904084 13471 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fb63ccf9e394100a3da0ba8aad34f19 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:03:14.904229 13476 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fb63ccf9e394100a3da0ba8aad34f19 [term 1 LEADER]: Becoming Leader. State: Replica: 9fb63ccf9e394100a3da0ba8aad34f19, State: Running, Role: LEADER
I20260929 19:03:14.904362 13476 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fb63ccf9e394100a3da0ba8aad34f19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fb63ccf9e394100a3da0ba8aad34f19" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37869 } }
I20260929 19:03:14.904724 13478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fb63ccf9e394100a3da0ba8aad34f19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fb63ccf9e394100a3da0ba8aad34f19. Latest consensus state: current_term: 1 leader_uuid: "9fb63ccf9e394100a3da0ba8aad34f19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fb63ccf9e394100a3da0ba8aad34f19" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37869 } } }
I20260929 19:03:14.904819 13478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fb63ccf9e394100a3da0ba8aad34f19 [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:14.905019 13485 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:03:14.905136 13477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fb63ccf9e394100a3da0ba8aad34f19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9fb63ccf9e394100a3da0ba8aad34f19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fb63ccf9e394100a3da0ba8aad34f19" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37869 } } }
I20260929 19:03:14.905232 13477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fb63ccf9e394100a3da0ba8aad34f19 [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:14.905527 13485 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:03:14.906999 13485 catalog_manager.cc:1347] Generated new cluster ID: ff211eceec6b4fd1be6151cdfd8c9612
I20260929 19:03:14.907055 13485 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:03:14.917569 13485 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:03:14.917711 13485 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:03:14.928342 13485 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9fb63ccf9e394100a3da0ba8aad34f19: Generated new TSK 0
I20260929 19:03:14.928496 13485 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:03:14.933364 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.65:0
--local_ip_for_outbound_sockets=127.8.158.65
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:37869
--builtin_ntp_servers=127.8.158.84:40419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 19:03:15.015733 13495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:15.015967 13495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:15.016005 13495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:15.017262 13495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:15.017328 13495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:15.017376 13495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.65
I20260929 19:03:15.018982 13495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:40419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:37869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.65
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:15.019254 13495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:15.019709 13495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:15.025905 13503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:15.025873 13500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:15.025866 13501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:15.026139 13495 server_base.cc:1034] running on GCE node
I20260929 19:03:15.026410 13495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:15.026748 13495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:15.027954 13495 hybrid_clock.cc:648] HybridClock initialized: now 1790708595027928 us; error 39 us; skew 500 ppm
I20260929 19:03:15.029569 13495 webserver.cc:460] Webserver started at http://127.8.158.65:33309/ using document root <none> and password file <none>
I20260929 19:03:15.029834 13495 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:15.029897 13495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:15.030026 13495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:15.031484 13495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-0/data/instance:
uuid: "17f859776ea34109bc74c9f356dda9ca"
format_stamp: "Formatted at 2026-09-29 19:03:15 on dist-test-slave-nmsz"
server_key: "29a0b472e8fde4a24fbd584bf45b3e12"
server_key_iv: "5714674e564bde6356ce7b974c20b709"
server_key_version: "encryptionkey@0"
I20260929 19:03:15.031891 13495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance:
uuid: "17f859776ea34109bc74c9f356dda9ca"
format_stamp: "Formatted at 2026-09-29 19:03:15 on dist-test-slave-nmsz"
server_key: "29a0b472e8fde4a24fbd584bf45b3e12"
server_key_iv: "5714674e564bde6356ce7b974c20b709"
server_key_version: "encryptionkey@0"
I20260929 19:03:15.033509 13495 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:03:15.034416 13509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:15.034585 13495 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:15.034724 13495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-0/wal
uuid: "17f859776ea34109bc74c9f356dda9ca"
format_stamp: "Formatted at 2026-09-29 19:03:15 on dist-test-slave-nmsz"
server_key: "29a0b472e8fde4a24fbd584bf45b3e12"
server_key_iv: "5714674e564bde6356ce7b974c20b709"
server_key_version: "encryptionkey@0"
I20260929 19:03:15.034813 13495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:15.037871 13495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:15.038152 13495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:15.038302 13495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:15.038560 13495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:15.038877 13495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:15.038932 13495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:15.038969 13495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:15.039000 13495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:15.044343 13495 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.65:42189
I20260929 19:03:15.044379 13622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.65:42189 every 8 connection(s)
I20260929 19:03:15.044698 13495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
I20260929 19:03:15.047535 13474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.045425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:52491 (local address 127.8.158.126:37869)
0929 19:03:15.045493 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:15.045495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:15.045688 (+ 193us) server_negotiation.cc:407] Connection header received
0929 19:03:15.046108 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:15.046109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:15.046113 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:15.046166 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:15.047297 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:15.047299 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:15.047299 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:03:15.047303 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:15.047316 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:03:15.047320 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:15.047370 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:15.047475 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 19:03:15.047487 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 19:03:15.047726 13625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.045378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:37869 (local address 127.8.158.65:52491)
0929 19:03:15.045572 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:15.045604 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:15.046027 (+ 423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:15.046217 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:15.046220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:15.046227 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:15.047159 (+ 932us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:03:15.047160 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:15.047185 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:03:15.047212 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:15.047406 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:15.047410 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:15.047413 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:03:15.047469 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 19:03:15.047513 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":95,"threads_started":1}
I20260929 19:03:15.048228 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 13495
I20260929 19:03:15.048328 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance
I20260929 19:03:15.048333 13623 heartbeater.cc:347] Connected to a master server at 127.8.158.126:37869
I20260929 19:03:15.048444 13623 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:15.048558 8825 external_mini_cluster.cc:1428] Setting key 038a9e58c2d7ce8865977261de711438
I20260929 19:03:15.048610 13623 heartbeater.cc:511] Master 127.8.158.126:37869 requested a full tablet report, sending...
I20260929 19:03:15.049079 13434 ts_manager.cc:194] Registered new tserver with Master: 17f859776ea34109bc74c9f356dda9ca (127.8.158.65:42189)
I20260929 19:03:15.049522 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.66:0
--local_ip_for_outbound_sockets=127.8.158.66
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:37869
--builtin_ntp_servers=127.8.158.84:40419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 19:03:15.128618 13626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:15.128811 13626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:15.128834 13626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:15.129910 13626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:15.129945 13626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:15.129971 13626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.66
I20260929 19:03:15.131327 13626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:40419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:37869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.66
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:15.131518 13626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:15.131753 13626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:15.136637 13632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:15.136610 13631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:15.136624 13634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:15.136919 13626 server_base.cc:1034] running on GCE node
I20260929 19:03:15.137137 13626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:15.137329 13626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:15.138464 13626 hybrid_clock.cc:648] HybridClock initialized: now 1790708595138449 us; error 38 us; skew 500 ppm
I20260929 19:03:15.139714 13626 webserver.cc:460] Webserver started at http://127.8.158.66:35783/ using document root <none> and password file <none>
I20260929 19:03:15.139945 13626 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:15.140027 13626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:15.140170 13626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:15.141196 13626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-1/data/instance:
uuid: "d56992188e7b4437bdcad8fb98970109"
format_stamp: "Formatted at 2026-09-29 19:03:15 on dist-test-slave-nmsz"
server_key: "f2c05a767fd8505ef1458ae0df1c72c9"
server_key_iv: "7992053bacbf3eb96a0de437b5e0e02d"
server_key_version: "encryptionkey@0"
I20260929 19:03:15.141539 13626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance:
uuid: "d56992188e7b4437bdcad8fb98970109"
format_stamp: "Formatted at 2026-09-29 19:03:15 on dist-test-slave-nmsz"
server_key: "f2c05a767fd8505ef1458ae0df1c72c9"
server_key_iv: "7992053bacbf3eb96a0de437b5e0e02d"
server_key_version: "encryptionkey@0"
I20260929 19:03:15.143033 13626 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:03:15.143769 13640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:15.143952 13626 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:03:15.144047 13626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-1/wal
uuid: "d56992188e7b4437bdcad8fb98970109"
format_stamp: "Formatted at 2026-09-29 19:03:15 on dist-test-slave-nmsz"
server_key: "f2c05a767fd8505ef1458ae0df1c72c9"
server_key_iv: "7992053bacbf3eb96a0de437b5e0e02d"
server_key_version: "encryptionkey@0"
I20260929 19:03:15.144124 13626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:15.147408 13626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:15.147657 13626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:15.147794 13626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:15.147990 13626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:15.148286 13626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:15.148339 13626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:15.148379 13626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:15.148406 13626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:15.153815 13626 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.66:41119
I20260929 19:03:15.153848 13753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.66:41119 every 8 connection(s)
I20260929 19:03:15.154175 13626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
I20260929 19:03:15.157217 13474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.154930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:50791 (local address 127.8.158.126:37869)
0929 19:03:15.154977 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:15.154979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:15.155256 (+ 277us) server_negotiation.cc:407] Connection header received
0929 19:03:15.155732 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:15.155733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:15.155739 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:15.155800 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:15.156926 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:15.156927 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:15.156927 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:03:15.156932 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:15.156947 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:03:15.156951 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:15.157024 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:15.157131 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 19:03:15.157145 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 19:03:15.157395 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.154930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:37869 (local address 127.8.158.66:50791)
0929 19:03:15.155140 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:15.155171 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:15.155629 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:15.155897 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:15.155900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:15.155907 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:15.156776 (+ 869us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:03:15.156776 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:15.156802 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:03:15.156828 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:15.157021 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:15.157026 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:15.157029 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:03:15.157140 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 19:03:15.157197 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":76,"threads_started":1}
I20260929 19:03:15.157893 13754 heartbeater.cc:347] Connected to a master server at 127.8.158.126:37869
I20260929 19:03:15.157964 13754 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:15.158124 13754 heartbeater.cc:511] Master 127.8.158.126:37869 requested a full tablet report, sending...
I20260929 19:03:15.158555 13434 ts_manager.cc:194] Registered new tserver with Master: d56992188e7b4437bdcad8fb98970109 (127.8.158.66:41119)
I20260929 19:03:15.163833 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 13626
I20260929 19:03:15.163920 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance
I20260929 19:03:15.164039 8825 external_mini_cluster.cc:1428] Setting key d8ea705c55f27a74db6fa0caf53658e3
I20260929 19:03:15.165056 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.67:0
--local_ip_for_outbound_sockets=127.8.158.67
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:37869
--builtin_ntp_servers=127.8.158.84:40419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 19:03:15.242717 13757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:15.242929 13757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:15.242965 13757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:15.244103 13757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:15.244161 13757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:15.244205 13757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.67
I20260929 19:03:15.245517 13757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:40419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:37869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.67
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:15.245764 13757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:15.246037 13757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:15.251055 13765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:15.251049 13762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:15.251151 13757 server_base.cc:1034] running on GCE node
W20260929 19:03:15.251161 13763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:15.251506 13757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:15.251696 13757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:15.252841 13757 hybrid_clock.cc:648] HybridClock initialized: now 1790708595252829 us; error 27 us; skew 500 ppm
I20260929 19:03:15.254091 13757 webserver.cc:460] Webserver started at http://127.8.158.67:35939/ using document root <none> and password file <none>
I20260929 19:03:15.254403 13757 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:15.254456 13757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:15.254577 13757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:15.255620 13757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-2/data/instance:
uuid: "e857b94975604ddab8353d78ed74bfea"
format_stamp: "Formatted at 2026-09-29 19:03:15 on dist-test-slave-nmsz"
server_key: "32f52d07af9927228836b0819b5224cc"
server_key_iv: "6dfb767f87374f06344d855b850a272c"
server_key_version: "encryptionkey@0"
I20260929 19:03:15.255960 13757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance:
uuid: "e857b94975604ddab8353d78ed74bfea"
format_stamp: "Formatted at 2026-09-29 19:03:15 on dist-test-slave-nmsz"
server_key: "32f52d07af9927228836b0819b5224cc"
server_key_iv: "6dfb767f87374f06344d855b850a272c"
server_key_version: "encryptionkey@0"
I20260929 19:03:15.257247 13757 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:03:15.258178 13771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:15.258388 13757 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:03:15.258481 13757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-2/wal
uuid: "e857b94975604ddab8353d78ed74bfea"
format_stamp: "Formatted at 2026-09-29 19:03:15 on dist-test-slave-nmsz"
server_key: "32f52d07af9927228836b0819b5224cc"
server_key_iv: "6dfb767f87374f06344d855b850a272c"
server_key_version: "encryptionkey@0"
I20260929 19:03:15.258539 13757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:15.261579 13757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:15.261762 13757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:15.261847 13757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:15.261997 13757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:15.262228 13757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:15.262256 13757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:15.262274 13757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:15.262287 13757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:15.267524 13757 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.67:40697
I20260929 19:03:15.267575 13884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.67:40697 every 8 connection(s)
I20260929 19:03:15.267908 13757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
I20260929 19:03:15.269196 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 13757
I20260929 19:03:15.269315 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance
I20260929 19:03:15.269528 8825 external_mini_cluster.cc:1428] Setting key 18df072d85b30d08a21c9aabb1780ee6
I20260929 19:03:15.271334 13474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.268776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:49039 (local address 127.8.158.126:37869)
0929 19:03:15.268857 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:15.268859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:15.269693 (+ 834us) server_negotiation.cc:407] Connection header received
0929 19:03:15.269733 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:15.269733 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:15.269739 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:15.269811 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:15.271159 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:15.271161 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:15.271161 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:03:15.271165 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:15.271179 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:03:15.271182 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:15.271226 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:15.271276 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 19:03:15.271287 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 19:03:15.271507 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.268743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:37869 (local address 127.8.158.67:49039)
0929 19:03:15.268920 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:15.268957 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:15.269522 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:15.269879 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:15.269884 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:15.269893 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:15.271029 (+ 1136us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:03:15.271029 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:15.271054 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:03:15.271080 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:15.271231 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:15.271234 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:15.271236 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 19:03:15.271283 (+ 47us) client_negotiation.cc:240] Negotiation successful
0929 19:03:15.271323 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
I20260929 19:03:15.271883 13885 heartbeater.cc:347] Connected to a master server at 127.8.158.126:37869
I20260929 19:03:15.271948 13885 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:15.272097 13885 heartbeater.cc:511] Master 127.8.158.126:37869 requested a full tablet report, sending...
I20260929 19:03:15.272392 13434 ts_manager.cc:194] Registered new tserver with Master: e857b94975604ddab8353d78ed74bfea (127.8.158.67:40697)
I20260929 19:03:15.281107 8825 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:03:15.283358 13474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.282466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56642 (local address 127.8.158.126:37869)
0929 19:03:15.282524 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:15.282526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:15.282604 (+ 78us) server_negotiation.cc:407] Connection header received
0929 19:03:15.282648 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:15.282649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:15.282653 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:15.282700 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:15.283201 (+ 501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:15.283202 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:15.283203 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:03:15.283206 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:15.283217 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:03:15.283221 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:15.283268 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:15.283301 (+ 33us) server_negotiation.cc:299] Negotiation successful
0929 19:03:15.283312 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 19:03:15.285220 13434 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56642:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:03:15.286880 13434 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:03:15.290979 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.289109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.66:41119 (local address 127.0.0.1:39886)
0929 19:03:15.289556 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:15.289585 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:15.289671 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:15.289966 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:15.289972 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:15.289979 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:15.290523 (+ 544us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:03:15.290524 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:15.290539 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:03:15.290562 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:15.290812 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:15.290819 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:15.290821 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 19:03:15.290883 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 19:03:15.290896 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":64,"threads_started":1}
I20260929 19:03:15.291178 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.289294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39886 (local address 127.8.158.66:41119)
0929 19:03:15.289480 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:15.289484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:15.289687 (+ 203us) server_negotiation.cc:407] Connection header received
0929 19:03:15.289721 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:15.289723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:15.289733 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:15.289827 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:15.290630 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:15.290636 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:15.290636 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:03:15.290640 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:15.290696 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:03:15.290720 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:15.290786 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:15.291089 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 19:03:15.291106 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":74,"threads_started":1}
I20260929 19:03:15.291208 13897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.289336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.67:40697 (local address 127.0.0.1:57324)
0929 19:03:15.289741 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:15.289749 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:15.289821 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:15.290204 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:15.290205 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:15.290208 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:15.290725 (+ 517us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:03:15.290725 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:15.290752 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:03:15.290765 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:15.290957 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:15.290958 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:15.290958 (+ 0us) client_negotiation.cc:769] Sending connection context
0929 19:03:15.291011 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 19:03:15.291024 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":62,"threads_started":1}
I20260929 19:03:15.291339 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.289203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47630 (local address 127.8.158.65:42189)
0929 19:03:15.289468 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:15.289474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:15.290124 (+ 650us) server_negotiation.cc:407] Connection header received
0929 19:03:15.290153 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:15.290155 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:15.290164 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:15.290255 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:15.291016 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:15.291023 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:15.291024 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:03:15.291028 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:15.291059 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:03:15.291082 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:15.291138 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:15.291245 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 19:03:15.291260 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":95,"threads_started":1}
I20260929 19:03:15.291504 13898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.289416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57324 (local address 127.8.158.67:40697)
0929 19:03:15.289576 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:15.289579 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:15.289820 (+ 241us) server_negotiation.cc:407] Connection header received
0929 19:03:15.289870 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:15.289872 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:15.289880 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:15.289951 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:15.290844 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:15.290849 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:15.290849 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:03:15.290854 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:15.290884 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:03:15.290906 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:15.290961 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:15.291197 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 19:03:15.291217 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":86,"threads_started":1}
I20260929 19:03:15.291323 13894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.289109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:42189 (local address 127.0.0.1:47630)
0929 19:03:15.289545 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:15.289585 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:15.289654 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:15.290282 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:15.290283 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:15.290286 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:15.290903 (+ 617us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:03:15.290903 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:15.290917 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:03:15.290929 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:15.291161 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:15.291162 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:15.291162 (+ 0us) client_negotiation.cc:769] Sending connection context
0929 19:03:15.291202 (+ 40us) client_negotiation.cc:240] Negotiation successful
0929 19:03:15.291214 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":2,"thread_start_us":72,"threads_started":1}
I20260929 19:03:15.291961 13688 tablet_service.cc:1467] Processing CreateTablet for tablet 21ff81c160ef49f2a757dc373155909c (DEFAULT_TABLE table=test-table [id=266048166e114272896d170b57cc90df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:15.292124 13819 tablet_service.cc:1467] Processing CreateTablet for tablet 21ff81c160ef49f2a757dc373155909c (DEFAULT_TABLE table=test-table [id=266048166e114272896d170b57cc90df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:15.292143 13557 tablet_service.cc:1467] Processing CreateTablet for tablet 21ff81c160ef49f2a757dc373155909c (DEFAULT_TABLE table=test-table [id=266048166e114272896d170b57cc90df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:15.292302 13688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21ff81c160ef49f2a757dc373155909c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:15.292356 13819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21ff81c160ef49f2a757dc373155909c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:15.292412 13557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21ff81c160ef49f2a757dc373155909c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:15.294919 13899 tablet_bootstrap.cc:492] T 21ff81c160ef49f2a757dc373155909c P e857b94975604ddab8353d78ed74bfea: Bootstrap starting.
I20260929 19:03:15.295457 13900 tablet_bootstrap.cc:492] T 21ff81c160ef49f2a757dc373155909c P d56992188e7b4437bdcad8fb98970109: Bootstrap starting.
I20260929 19:03:15.295804 13899 tablet_bootstrap.cc:654] T 21ff81c160ef49f2a757dc373155909c P e857b94975604ddab8353d78ed74bfea: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:15.296070 13900 tablet_bootstrap.cc:654] T 21ff81c160ef49f2a757dc373155909c P d56992188e7b4437bdcad8fb98970109: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:15.296106 13899 log.cc:824] T 21ff81c160ef49f2a757dc373155909c P e857b94975604ddab8353d78ed74bfea: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:15.296394 13900 log.cc:824] T 21ff81c160ef49f2a757dc373155909c P d56992188e7b4437bdcad8fb98970109: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:15.296445 13901 tablet_bootstrap.cc:492] T 21ff81c160ef49f2a757dc373155909c P 17f859776ea34109bc74c9f356dda9ca: Bootstrap starting.
I20260929 19:03:15.296888 13899 tablet_bootstrap.cc:492] T 21ff81c160ef49f2a757dc373155909c P e857b94975604ddab8353d78ed74bfea: No bootstrap required, opened a new log
I20260929 19:03:15.296957 13899 ts_tablet_manager.cc:1397] T 21ff81c160ef49f2a757dc373155909c P e857b94975604ddab8353d78ed74bfea: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:03:15.297113 13900 tablet_bootstrap.cc:492] T 21ff81c160ef49f2a757dc373155909c P d56992188e7b4437bdcad8fb98970109: No bootstrap required, opened a new log
I20260929 19:03:15.297127 13901 tablet_bootstrap.cc:654] T 21ff81c160ef49f2a757dc373155909c P 17f859776ea34109bc74c9f356dda9ca: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:15.297170 13900 ts_tablet_manager.cc:1397] T 21ff81c160ef49f2a757dc373155909c P d56992188e7b4437bdcad8fb98970109: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:03:15.297428 13901 log.cc:824] T 21ff81c160ef49f2a757dc373155909c P 17f859776ea34109bc74c9f356dda9ca: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:15.298168 13901 tablet_bootstrap.cc:492] T 21ff81c160ef49f2a757dc373155909c P 17f859776ea34109bc74c9f356dda9ca: No bootstrap required, opened a new log
I20260929 19:03:15.298218 13901 ts_tablet_manager.cc:1397] T 21ff81c160ef49f2a757dc373155909c P 17f859776ea34109bc74c9f356dda9ca: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:03:15.298981 13899 raft_consensus.cc:348] T 21ff81c160ef49f2a757dc373155909c P e857b94975604ddab8353d78ed74bfea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f859776ea34109bc74c9f356dda9ca" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 42189 } } peers { permanent_uuid: "d56992188e7b4437bdcad8fb98970109" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 41119 } } peers { permanent_uuid: "e857b94975604ddab8353d78ed74bfea" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 40697 } }
I20260929 19:03:15.299093 13899 raft_consensus.cc:374] T 21ff81c160ef49f2a757dc373155909c P e857b94975604ddab8353d78ed74bfea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:15.299113 13899 raft_consensus.cc:729] T 21ff81c160ef49f2a757dc373155909c P e857b94975604ddab8353d78ed74bfea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e857b94975604ddab8353d78ed74bfea, State: Initialized, Role: FOLLOWER
I20260929 19:03:15.299192 13899 consensus_queue.cc:260] T 21ff81c160ef49f2a757dc373155909c P e857b94975604ddab8353d78ed74bfea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f859776ea34109bc74c9f356dda9ca" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 42189 } } peers { permanent_uuid: "d56992188e7b4437bdcad8fb98970109" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 41119 } } peers { permanent_uuid: "e857b94975604ddab8353d78ed74bfea" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 40697 } }
I20260929 19:03:15.299360 13899 ts_tablet_manager.cc:1428] T 21ff81c160ef49f2a757dc373155909c P e857b94975604ddab8353d78ed74bfea: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:03:15.299459 13885 heartbeater.cc:503] Master 127.8.158.126:37869 was elected leader, sending a full tablet report...
I20260929 19:03:15.300004 13900 raft_consensus.cc:348] T 21ff81c160ef49f2a757dc373155909c P d56992188e7b4437bdcad8fb98970109 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f859776ea34109bc74c9f356dda9ca" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 42189 } } peers { permanent_uuid: "d56992188e7b4437bdcad8fb98970109" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 41119 } } peers { permanent_uuid: "e857b94975604ddab8353d78ed74bfea" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 40697 } }
I20260929 19:03:15.300267 13900 raft_consensus.cc:374] T 21ff81c160ef49f2a757dc373155909c P d56992188e7b4437bdcad8fb98970109 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:15.300346 13900 raft_consensus.cc:729] T 21ff81c160ef49f2a757dc373155909c P d56992188e7b4437bdcad8fb98970109 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d56992188e7b4437bdcad8fb98970109, State: Initialized, Role: FOLLOWER
I20260929 19:03:15.300575 13900 consensus_queue.cc:260] T 21ff81c160ef49f2a757dc373155909c P d56992188e7b4437bdcad8fb98970109 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f859776ea34109bc74c9f356dda9ca" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 42189 } } peers { permanent_uuid: "d56992188e7b4437bdcad8fb98970109" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 41119 } } peers { permanent_uuid: "e857b94975604ddab8353d78ed74bfea" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 40697 } }
I20260929 19:03:15.300823 13900 ts_tablet_manager.cc:1428] T 21ff81c160ef49f2a757dc373155909c P d56992188e7b4437bdcad8fb98970109: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260929 19:03:15.300782 13901 raft_consensus.cc:348] T 21ff81c160ef49f2a757dc373155909c P 17f859776ea34109bc74c9f356dda9ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f859776ea34109bc74c9f356dda9ca" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 42189 } } peers { permanent_uuid: "d56992188e7b4437bdcad8fb98970109" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 41119 } } peers { permanent_uuid: "e857b94975604ddab8353d78ed74bfea" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 40697 } }
I20260929 19:03:15.300927 13901 raft_consensus.cc:374] T 21ff81c160ef49f2a757dc373155909c P 17f859776ea34109bc74c9f356dda9ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:15.300944 13754 heartbeater.cc:503] Master 127.8.158.126:37869 was elected leader, sending a full tablet report...
I20260929 19:03:15.300973 13901 raft_consensus.cc:729] T 21ff81c160ef49f2a757dc373155909c P 17f859776ea34109bc74c9f356dda9ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17f859776ea34109bc74c9f356dda9ca, State: Initialized, Role: FOLLOWER
I20260929 19:03:15.301069 13901 consensus_queue.cc:260] T 21ff81c160ef49f2a757dc373155909c P 17f859776ea34109bc74c9f356dda9ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f859776ea34109bc74c9f356dda9ca" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 42189 } } peers { permanent_uuid: "d56992188e7b4437bdcad8fb98970109" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 41119 } } peers { permanent_uuid: "e857b94975604ddab8353d78ed74bfea" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 40697 } }
I20260929 19:03:15.301307 13901 ts_tablet_manager.cc:1428] T 21ff81c160ef49f2a757dc373155909c P 17f859776ea34109bc74c9f356dda9ca: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260929 19:03:15.301473 13623 heartbeater.cc:503] Master 127.8.158.126:37869 was elected leader, sending a full tablet report...
I20260929 19:03:15.318466 13907 raft_consensus.cc:482] T 21ff81c160ef49f2a757dc373155909c P 17f859776ea34109bc74c9f356dda9ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:03:15.318617 13907 raft_consensus.cc:504] T 21ff81c160ef49f2a757dc373155909c P 17f859776ea34109bc74c9f356dda9ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f859776ea34109bc74c9f356dda9ca" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 42189 } } peers { permanent_uuid: "d56992188e7b4437bdcad8fb98970109" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 41119 } } peers { permanent_uuid: "e857b94975604ddab8353d78ed74bfea" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 40697 } }
I20260929 19:03:15.319024 13907 leader_election.cc:288] T 21ff81c160ef49f2a757dc373155909c P 17f859776ea34109bc74c9f356dda9ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d56992188e7b4437bdcad8fb98970109 (127.8.158.66:41119), e857b94975604ddab8353d78ed74bfea (127.8.158.67:40697)
I20260929 19:03:15.320379 13625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.319192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.67:40697 (local address 127.8.158.65:34419)
0929 19:03:15.319255 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:15.319267 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:15.319307 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:15.319518 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:15.319520 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:15.319522 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:15.319990 (+ 468us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:03:15.319991 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:15.319997 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:03:15.320009 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:15.320214 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:15.320215 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:15.320216 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 19:03:15.320275 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 19:03:15.320289 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 19:03:15.320463 13898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.319284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:34419 (local address 127.8.158.67:40697)
0929 19:03:15.319333 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:15.319335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:15.319346 (+ 11us) server_negotiation.cc:407] Connection header received
0929 19:03:15.319373 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:15.319374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:15.319380 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:15.319497 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:15.320115 (+ 618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:15.320116 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:15.320117 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:03:15.320121 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:15.320135 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:03:15.320140 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:15.320216 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:15.320302 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 19:03:15.320318 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 19:03:15.320504 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.319282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:52427 (local address 127.8.158.66:41119)
0929 19:03:15.319327 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:15.319329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:15.319523 (+ 194us) server_negotiation.cc:407] Connection header received
0929 19:03:15.319553 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:15.319554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:15.319561 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:15.319621 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:15.320265 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:15.320266 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:15.320266 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:03:15.320271 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:15.320288 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:03:15.320292 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:15.320350 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:15.320404 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 19:03:15.320418 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 19:03:15.320746 13908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.319193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.66:41119 (local address 127.8.158.65:52427)
0929 19:03:15.319402 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:15.319413 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:15.319482 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:15.319629 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:15.319630 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:15.319632 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:15.320178 (+ 546us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:03:15.320179 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:15.320185 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:03:15.320197 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:15.320353 (+ 156us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:15.320354 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:15.320355 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 19:03:15.320404 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 19:03:15.320416 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":21,"thread_start_us":110,"threads_started":1}
I20260929 19:03:15.320940 13839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21ff81c160ef49f2a757dc373155909c" candidate_uuid: "17f859776ea34109bc74c9f356dda9ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e857b94975604ddab8353d78ed74bfea" is_pre_election: true
I20260929 19:03:15.321087 13839 raft_consensus.cc:2454] T 21ff81c160ef49f2a757dc373155909c P e857b94975604ddab8353d78ed74bfea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17f859776ea34109bc74c9f356dda9ca in term 0.
I20260929 19:03:15.321098 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21ff81c160ef49f2a757dc373155909c" candidate_uuid: "17f859776ea34109bc74c9f356dda9ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d56992188e7b4437bdcad8fb98970109" is_pre_election: true
I20260929 19:03:15.321247 13708 raft_consensus.cc:2454] T 21ff81c160ef49f2a757dc373155909c P d56992188e7b4437bdcad8fb98970109 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17f859776ea34109bc74c9f356dda9ca in term 0.
I20260929 19:03:15.321307 13511 leader_election.cc:302] T 21ff81c160ef49f2a757dc373155909c P 17f859776ea34109bc74c9f356dda9ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17f859776ea34109bc74c9f356dda9ca, e857b94975604ddab8353d78ed74bfea; no voters:
I20260929 19:03:15.321426 13907 raft_consensus.cc:2781] T 21ff81c160ef49f2a757dc373155909c P 17f859776ea34109bc74c9f356dda9ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:03:15.321477 13907 raft_consensus.cc:482] T 21ff81c160ef49f2a757dc373155909c P 17f859776ea34109bc74c9f356dda9ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:03:15.321506 13907 raft_consensus.cc:3037] T 21ff81c160ef49f2a757dc373155909c P 17f859776ea34109bc74c9f356dda9ca [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:15.322199 13907 raft_consensus.cc:504] T 21ff81c160ef49f2a757dc373155909c P 17f859776ea34109bc74c9f356dda9ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f859776ea34109bc74c9f356dda9ca" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 42189 } } peers { permanent_uuid: "d56992188e7b4437bdcad8fb98970109" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 41119 } } peers { permanent_uuid: "e857b94975604ddab8353d78ed74bfea" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 40697 } }
I20260929 19:03:15.322360 13907 leader_election.cc:288] T 21ff81c160ef49f2a757dc373155909c P 17f859776ea34109bc74c9f356dda9ca [CANDIDATE]: Term 1 election: Requested vote from peers d56992188e7b4437bdcad8fb98970109 (127.8.158.66:41119), e857b94975604ddab8353d78ed74bfea (127.8.158.67:40697)
I20260929 19:03:15.322487 13839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21ff81c160ef49f2a757dc373155909c" candidate_uuid: "17f859776ea34109bc74c9f356dda9ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e857b94975604ddab8353d78ed74bfea"
I20260929 19:03:15.322525 13708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21ff81c160ef49f2a757dc373155909c" candidate_uuid: "17f859776ea34109bc74c9f356dda9ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d56992188e7b4437bdcad8fb98970109"
I20260929 19:03:15.322582 13839 raft_consensus.cc:3037] T 21ff81c160ef49f2a757dc373155909c P e857b94975604ddab8353d78ed74bfea [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:15.322600 13708 raft_consensus.cc:3037] T 21ff81c160ef49f2a757dc373155909c P d56992188e7b4437bdcad8fb98970109 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:15.323241 13839 raft_consensus.cc:2454] T 21ff81c160ef49f2a757dc373155909c P e857b94975604ddab8353d78ed74bfea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17f859776ea34109bc74c9f356dda9ca in term 1.
I20260929 19:03:15.323349 13708 raft_consensus.cc:2454] T 21ff81c160ef49f2a757dc373155909c P d56992188e7b4437bdcad8fb98970109 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17f859776ea34109bc74c9f356dda9ca in term 1.
I20260929 19:03:15.323457 13511 leader_election.cc:302] T 21ff81c160ef49f2a757dc373155909c P 17f859776ea34109bc74c9f356dda9ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17f859776ea34109bc74c9f356dda9ca, e857b94975604ddab8353d78ed74bfea; no voters:
I20260929 19:03:15.323536 13907 raft_consensus.cc:2781] T 21ff81c160ef49f2a757dc373155909c P 17f859776ea34109bc74c9f356dda9ca [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:03:15.323699 13907 raft_consensus.cc:686] T 21ff81c160ef49f2a757dc373155909c P 17f859776ea34109bc74c9f356dda9ca [term 1 LEADER]: Becoming Leader. State: Replica: 17f859776ea34109bc74c9f356dda9ca, State: Running, Role: LEADER
I20260929 19:03:15.323796 13907 consensus_queue.cc:237] T 21ff81c160ef49f2a757dc373155909c P 17f859776ea34109bc74c9f356dda9ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f859776ea34109bc74c9f356dda9ca" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 42189 } } peers { permanent_uuid: "d56992188e7b4437bdcad8fb98970109" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 41119 } } peers { permanent_uuid: "e857b94975604ddab8353d78ed74bfea" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 40697 } }
I20260929 19:03:15.324501 13434 catalog_manager.cc:5495] T 21ff81c160ef49f2a757dc373155909c P 17f859776ea34109bc74c9f356dda9ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 17f859776ea34109bc74c9f356dda9ca (127.8.158.65). New cstate: current_term: 1 leader_uuid: "17f859776ea34109bc74c9f356dda9ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f859776ea34109bc74c9f356dda9ca" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 42189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d56992188e7b4437bdcad8fb98970109" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 41119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e857b94975604ddab8353d78ed74bfea" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 40697 } health_report { overall_health: UNKNOWN } } }
I20260929 19:03:15.336812 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.335860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47646 (local address 127.8.158.65:42189)
0929 19:03:15.335897 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:15.335899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:15.335907 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:03:15.335985 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:15.335986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:15.336004 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:15.336070 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:15.336595 (+ 525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:15.336596 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:15.336597 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:03:15.336601 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:15.336615 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:03:15.336619 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:15.336676 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:15.336731 (+ 55us) server_negotiation.cc:299] Negotiation successful
0929 19:03:15.336744 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 19:03:15.338434 13708 raft_consensus.cc:1261] T 21ff81c160ef49f2a757dc373155909c P d56992188e7b4437bdcad8fb98970109 [term 1 FOLLOWER]: Refusing update from remote peer 17f859776ea34109bc74c9f356dda9ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:03:15.338483 13839 raft_consensus.cc:1261] T 21ff81c160ef49f2a757dc373155909c P e857b94975604ddab8353d78ed74bfea [term 1 FOLLOWER]: Refusing update from remote peer 17f859776ea34109bc74c9f356dda9ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:03:15.338824 13907 consensus_queue.cc:1035] T 21ff81c160ef49f2a757dc373155909c P 17f859776ea34109bc74c9f356dda9ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d56992188e7b4437bdcad8fb98970109" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:03:15.338959 13909 consensus_queue.cc:1035] T 21ff81c160ef49f2a757dc373155909c P 17f859776ea34109bc74c9f356dda9ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "e857b94975604ddab8353d78ed74bfea" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 40697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:03:15.340965 13912 mvcc.cc:205] Tried to move back new op lower bound from 7334742406503903232 to 7334742406446891008. Current Snapshot: MvccSnapshot[applied={T|T < 7334742406503903232}]
I20260929 19:03:15.341557 13913 mvcc.cc:205] Tried to move back new op lower bound from 7334742406503903232 to 7334742406446891008. Current Snapshot: MvccSnapshot[applied={T|T < 7334742406503903232}]
I20260929 19:03:15.343139 13435 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:03:15.343230 13435 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:03:15.343957 13435 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9fb63ccf9e394100a3da0ba8aad34f19: Sending DeleteTablet for 3 replicas of tablet 21ff81c160ef49f2a757dc373155909c
I20260929 19:03:15.344304 13819 tablet_service.cc:1514] Processing DeleteTablet for tablet 21ff81c160ef49f2a757dc373155909c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:15 UTC) from {username='slave'} at 127.0.0.1:57324
I20260929 19:03:15.344300 13557 tablet_service.cc:1514] Processing DeleteTablet for tablet 21ff81c160ef49f2a757dc373155909c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:15 UTC) from {username='slave'} at 127.0.0.1:47630
I20260929 19:03:15.344658 13922 tablet_replica.cc:330] T 21ff81c160ef49f2a757dc373155909c P 17f859776ea34109bc74c9f356dda9ca: stopping tablet replica
I20260929 19:03:15.344599 13688 tablet_service.cc:1514] Processing DeleteTablet for tablet 21ff81c160ef49f2a757dc373155909c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:15 UTC) from {username='slave'} at 127.0.0.1:39886
I20260929 19:03:15.344813 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 13495
I20260929 19:03:15.345536 13923 tablet_replica.cc:330] T 21ff81c160ef49f2a757dc373155909c P e857b94975604ddab8353d78ed74bfea: stopping tablet replica
I20260929 19:03:15.345611 13923 raft_consensus.cc:2229] T 21ff81c160ef49f2a757dc373155909c P e857b94975604ddab8353d78ed74bfea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:03:15.345675 13923 raft_consensus.cc:2258] T 21ff81c160ef49f2a757dc373155909c P e857b94975604ddab8353d78ed74bfea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:03:15.346041 13923 ts_tablet_manager.cc:1905] T 21ff81c160ef49f2a757dc373155909c P e857b94975604ddab8353d78ed74bfea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:03:15.347464 13924 tablet_replica.cc:330] T 21ff81c160ef49f2a757dc373155909c P d56992188e7b4437bdcad8fb98970109: stopping tablet replica
I20260929 19:03:15.347545 13924 raft_consensus.cc:2229] T 21ff81c160ef49f2a757dc373155909c P d56992188e7b4437bdcad8fb98970109 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:03:15.347697 13924 raft_consensus.cc:2258] T 21ff81c160ef49f2a757dc373155909c P d56992188e7b4437bdcad8fb98970109 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:03:15.347810 13923 ts_tablet_manager.cc:1918] T 21ff81c160ef49f2a757dc373155909c P e857b94975604ddab8353d78ed74bfea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:03:15.347880 13923 log.cc:1196] T 21ff81c160ef49f2a757dc373155909c P e857b94975604ddab8353d78ed74bfea: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-2/wal/wals/21ff81c160ef49f2a757dc373155909c
I20260929 19:03:15.348091 13924 ts_tablet_manager.cc:1905] T 21ff81c160ef49f2a757dc373155909c P d56992188e7b4437bdcad8fb98970109: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:03:15.348163 13923 ts_tablet_manager.cc:1939] T 21ff81c160ef49f2a757dc373155909c P e857b94975604ddab8353d78ed74bfea: Deleting consensus metadata
I20260929 19:03:15.348608 13420 catalog_manager.cc:4841] TS e857b94975604ddab8353d78ed74bfea (127.8.158.67:40697): tablet 21ff81c160ef49f2a757dc373155909c (table test-table [id=266048166e114272896d170b57cc90df]) successfully deleted
I20260929 19:03:15.349555 13924 ts_tablet_manager.cc:1918] T 21ff81c160ef49f2a757dc373155909c P d56992188e7b4437bdcad8fb98970109: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:03:15.349617 13924 log.cc:1196] T 21ff81c160ef49f2a757dc373155909c P d56992188e7b4437bdcad8fb98970109: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708573736807-8825-0/minicluster-data/ts-1/wal/wals/21ff81c160ef49f2a757dc373155909c
I20260929 19:03:15.349848 13924 ts_tablet_manager.cc:1939] T 21ff81c160ef49f2a757dc373155909c P d56992188e7b4437bdcad8fb98970109: Deleting consensus metadata
I20260929 19:03:15.350325 13421 catalog_manager.cc:4841] TS d56992188e7b4437bdcad8fb98970109 (127.8.158.66:41119): tablet 21ff81c160ef49f2a757dc373155909c (table test-table [id=266048166e114272896d170b57cc90df]) successfully deleted
W20260929 19:03:15.350379 13420 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.158.65:42189 (error 108)
I20260929 19:03:15.350584 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 13626
I20260929 19:03:15.351145 13894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.350716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:42189 (local address 127.0.0.1:47652)
0929 19:03:15.351076 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:15.351114 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.65:42189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344}
W20260929 19:03:15.351270 13420 catalog_manager.cc:4584] TS 17f859776ea34109bc74c9f356dda9ca (127.8.158.65:42189): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 21ff81c160ef49f2a757dc373155909c: Network error: Client connection negotiation failed: client connection to 127.8.158.65:42189: connect: Connection refused (error 111)
I20260929 19:03:15.355589 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 13757
I20260929 19:03:15.359529 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 13404
2026-09-29T19:03:15Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (591 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 19:03:15.378494 8825 mini_oidc.cc:152] Starting JWKS server
I20260929 19:03:15.379555 8825 webserver.cc:460] Webserver started at https://127.0.0.1:40159/ using document root <none> and password file <none>
I20260929 19:03:15.379655 8825 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 19:03:15.379841 8825 webserver.cc:460] Webserver started at http://127.0.0.1:41891/ using document root <none> and password file <none>
2026-09-29T19:03:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:03:15Z Disabled control of system clock
I20260929 19:03:15.387369 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:34589
--webserver_interface=127.8.158.126
--webserver_port=0
--builtin_ntp_servers=127.8.158.84:45053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:34589
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40159/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708573736807-8825-0/testchainca.cert with env {}
W20260929 19:03:15.465237 13931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:15.465421 13931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:15.465443 13931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:15.466563 13931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:03:15.466632 13931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:15.466646 13931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:15.466660 13931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:03:15.466671 13931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 19:03:15.466683 13931 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 19:03:15.466694 13931 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40159/jwks.json
I20260929 19:03:15.468142 13931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:45053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:34589
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:34589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40159/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.158.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708573736807-8825-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:15.468619 13931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:15.468808 13931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:15.473569 13936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:15.473569 13937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:15.473579 13939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:15.473806 13931 server_base.cc:1034] running on GCE node
I20260929 19:03:15.474069 13931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:15.474298 13931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:15.475497 13931 hybrid_clock.cc:648] HybridClock initialized: now 1790708595475477 us; error 41 us; skew 500 ppm
I20260929 19:03:15.476810 13931 webserver.cc:460] Webserver started at http://127.8.158.126:39261/ using document root <none> and password file <none>
I20260929 19:03:15.477057 13931 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:15.477129 13931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:15.477275 13931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:15.478292 13931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "0ab7b873fcca45a289f2c27257fc5f8a"
format_stamp: "Formatted at 2026-09-29 19:03:15 on dist-test-slave-nmsz"
server_key: "1dc53b0491c22abb4945048119dcc308"
server_key_iv: "00813fdaa2314b97888887529de3e146"
server_key_version: "encryptionkey@0"
I20260929 19:03:15.478713 13931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "0ab7b873fcca45a289f2c27257fc5f8a"
format_stamp: "Formatted at 2026-09-29 19:03:15 on dist-test-slave-nmsz"
server_key: "1dc53b0491c22abb4945048119dcc308"
server_key_iv: "00813fdaa2314b97888887529de3e146"
server_key_version: "encryptionkey@0"
I20260929 19:03:15.479947 13931 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:15.480630 13945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:15.480798 13931 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:03:15.480899 13931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "0ab7b873fcca45a289f2c27257fc5f8a"
format_stamp: "Formatted at 2026-09-29 19:03:15 on dist-test-slave-nmsz"
server_key: "1dc53b0491c22abb4945048119dcc308"
server_key_iv: "00813fdaa2314b97888887529de3e146"
server_key_version: "encryptionkey@0"
I20260929 19:03:15.480971 13931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:15.492252 13931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:15.492524 13931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:15.492682 13931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:15.496613 13931 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.126:34589
I20260929 19:03:15.496640 13997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.126:34589 every 8 connection(s)
I20260929 19:03:15.496955 13931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:03:15.497854 13998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:15.500016 13998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ab7b873fcca45a289f2c27257fc5f8a: Bootstrap starting.
I20260929 19:03:15.500528 13998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ab7b873fcca45a289f2c27257fc5f8a: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:15.500752 13998 log.cc:824] T 00000000000000000000000000000000 P 0ab7b873fcca45a289f2c27257fc5f8a: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:15.501338 13998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ab7b873fcca45a289f2c27257fc5f8a: No bootstrap required, opened a new log
I20260929 19:03:15.501348 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 13931
I20260929 19:03:15.501461 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:03:15.501631 8825 external_mini_cluster.cc:1428] Setting key 37ef112ebbe80091636f2eab33f6e922
I20260929 19:03:15.503265 13998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ab7b873fcca45a289f2c27257fc5f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab7b873fcca45a289f2c27257fc5f8a" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 34589 } }
I20260929 19:03:15.503423 13998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ab7b873fcca45a289f2c27257fc5f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:15.503468 13998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ab7b873fcca45a289f2c27257fc5f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ab7b873fcca45a289f2c27257fc5f8a, State: Initialized, Role: FOLLOWER
I20260929 19:03:15.503594 13998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ab7b873fcca45a289f2c27257fc5f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab7b873fcca45a289f2c27257fc5f8a" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 34589 } }
I20260929 19:03:15.503690 13998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ab7b873fcca45a289f2c27257fc5f8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:03:15.503727 13998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ab7b873fcca45a289f2c27257fc5f8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:03:15.503772 13998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ab7b873fcca45a289f2c27257fc5f8a [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:15.504261 13998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ab7b873fcca45a289f2c27257fc5f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab7b873fcca45a289f2c27257fc5f8a" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 34589 } }
I20260929 19:03:15.504375 13998 leader_election.cc:302] T 00000000000000000000000000000000 P 0ab7b873fcca45a289f2c27257fc5f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ab7b873fcca45a289f2c27257fc5f8a; no voters:
I20260929 19:03:15.504523 13998 leader_election.cc:288] T 00000000000000000000000000000000 P 0ab7b873fcca45a289f2c27257fc5f8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:03:15.504575 14003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ab7b873fcca45a289f2c27257fc5f8a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:03:15.504709 14003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ab7b873fcca45a289f2c27257fc5f8a [term 1 LEADER]: Becoming Leader. State: Replica: 0ab7b873fcca45a289f2c27257fc5f8a, State: Running, Role: LEADER
I20260929 19:03:15.504778 13998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ab7b873fcca45a289f2c27257fc5f8a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:03:15.504787 14003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ab7b873fcca45a289f2c27257fc5f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab7b873fcca45a289f2c27257fc5f8a" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 34589 } }
I20260929 19:03:15.505133 14003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ab7b873fcca45a289f2c27257fc5f8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ab7b873fcca45a289f2c27257fc5f8a. Latest consensus state: current_term: 1 leader_uuid: "0ab7b873fcca45a289f2c27257fc5f8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab7b873fcca45a289f2c27257fc5f8a" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 34589 } } }
I20260929 19:03:15.505230 14003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ab7b873fcca45a289f2c27257fc5f8a [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:15.505146 14005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ab7b873fcca45a289f2c27257fc5f8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ab7b873fcca45a289f2c27257fc5f8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab7b873fcca45a289f2c27257fc5f8a" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 34589 } } }
I20260929 19:03:15.505401 14005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ab7b873fcca45a289f2c27257fc5f8a [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:15.505450 14010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:03:15.505957 14010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:03:15.507737 14010 catalog_manager.cc:1347] Generated new cluster ID: 35eb6c44d94f41e9ad122d54116c2d22
I20260929 19:03:15.507792 14010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:03:15.508641 14002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.502131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53918 (local address 127.8.158.126:34589)
0929 19:03:15.502384 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:15.502394 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:15.502426 (+ 32us) server_negotiation.cc:407] Connection header received
0929 19:03:15.503255 (+ 829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:15.503262 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:15.503318 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:15.503480 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:15.503719 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.505238 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.506434 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.507064 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.507186 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:15.507928 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:15.507932 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:15.507932 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:03:15.508134 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:15.508172 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:03:15.508184 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:15.508242 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:15.508372 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 19:03:15.508444 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":75,"threads_started":1}
I20260929 19:03:15.509516 8825 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:03:15.516273 14010 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:03:15.516752 14010 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:03:15.521755 14010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ab7b873fcca45a289f2c27257fc5f8a: Generated new TSK 0
I20260929 19:03:15.521910 14010 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 19:03:15.550480 14028 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 19:03:15.550587 14028 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 19:03:15.550798 14002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.535857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53930 (local address 127.8.158.126:34589)
0929 19:03:15.535913 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:15.535915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:15.536047 (+ 132us) server_negotiation.cc:407] Connection header received
0929 19:03:15.536143 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:15.536143 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:15.536154 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:15.536210 (+ 56us) server_negotiation.cc:226] Negotiated authn=JWT
0929 19:03:15.536373 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.536829 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.537928 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.538112 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.538166 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:15.538406 (+ 240us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 19:03:15.550588 (+ 12182us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40159/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 19:03:15.550737 (+ 149us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53930: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40159/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":32}
W20260929 19:03:15.550858 14002 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53930: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40159/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 19:03:15.551128 14027 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:15.535788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:34589 (local address 127.0.0.1:53930)
0929 19:03:15.535973 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:15.535984 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:15.536056 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:15.536238 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:15.536238 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:15.536240 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0929 19:03:15.536307 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:15.536309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.536900 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.536901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:15.537843 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:15.537845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.538186 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.538187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:15.538252 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:15.538272 (+ 20us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 19:03:15.550755 (+ 12483us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40159/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 19:03:15.550849 (+ 94us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.158.126:34589: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40159/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":95,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.8.158.126:34589: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40159/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 19:03:15.551540 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 13931
2026-09-29T19:03:15Z chronyd exiting
I20260929 19:03:15.587297 8825 test_util.cc:175] -----------------------------------------------
I20260929 19:03:15.587394 8825 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708573736807-8825-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (209 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 19:03:15.588182 8825 mini_oidc.cc:152] Starting JWKS server
I20260929 19:03:15.588861 8825 webserver.cc:460] Webserver started at https://127.0.0.1:38699/ using document root <none> and password file <none>
I20260929 19:03:15.588929 8825 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 19:03:15.589108 8825 webserver.cc:460] Webserver started at http://127.0.0.1:45781/ using document root <none> and password file <none>
2026-09-29T19:03:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:03:15Z Disabled control of system clock
I20260929 19:03:15.596122 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:44355
--webserver_interface=127.8.158.126
--webserver_port=0
--builtin_ntp_servers=127.8.158.84:37627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:44355
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38699/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708573736807-8825-0/testchainca.cert with env {}
W20260929 19:03:15.673130 14034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:15.673319 14034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:15.673341 14034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:15.674444 14034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:03:15.674479 14034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:15.674491 14034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:15.674503 14034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:03:15.674515 14034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 19:03:15.674527 14034 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 19:03:15.674539 14034 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38699/jwks.json
I20260929 19:03:15.676000 14034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:37627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:44355
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:44355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38699/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.158.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708573736807-8825-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:15.676183 14034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:15.676337 14034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:15.681028 14039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:15.681151 14042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:15.681056 14040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:15.681233 14034 server_base.cc:1034] running on GCE node
I20260929 19:03:15.681589 14034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:15.681859 14034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:15.683022 14034 hybrid_clock.cc:648] HybridClock initialized: now 1790708595682996 us; error 42 us; skew 500 ppm
I20260929 19:03:15.684240 14034 webserver.cc:460] Webserver started at http://127.8.158.126:40835/ using document root <none> and password file <none>
I20260929 19:03:15.684480 14034 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:15.684549 14034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:15.684715 14034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:15.685707 14034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "2e45f7c5415744bf9f33ddc87fd59771"
format_stamp: "Formatted at 2026-09-29 19:03:15 on dist-test-slave-nmsz"
server_key: "5af1a69d0947336b13c1bb5014a05c28"
server_key_iv: "34b187690f42a3c74e06629fe0d7eaa5"
server_key_version: "encryptionkey@0"
I20260929 19:03:15.686053 14034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "2e45f7c5415744bf9f33ddc87fd59771"
format_stamp: "Formatted at 2026-09-29 19:03:15 on dist-test-slave-nmsz"
server_key: "5af1a69d0947336b13c1bb5014a05c28"
server_key_iv: "34b187690f42a3c74e06629fe0d7eaa5"
server_key_version: "encryptionkey@0"
I20260929 19:03:15.687420 14034 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:03:15.688104 14048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:15.688272 14034 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:03:15.688369 14034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "2e45f7c5415744bf9f33ddc87fd59771"
format_stamp: "Formatted at 2026-09-29 19:03:15 on dist-test-slave-nmsz"
server_key: "5af1a69d0947336b13c1bb5014a05c28"
server_key_iv: "34b187690f42a3c74e06629fe0d7eaa5"
server_key_version: "encryptionkey@0"
I20260929 19:03:15.688455 14034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:15.711673 14034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:15.711962 14034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:15.712118 14034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:15.715934 14034 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.126:44355
I20260929 19:03:15.715972 14100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.126:44355 every 8 connection(s)
I20260929 19:03:15.716318 14034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:03:15.717216 14101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:15.719403 14101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e45f7c5415744bf9f33ddc87fd59771: Bootstrap starting.
I20260929 19:03:15.720012 14101 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e45f7c5415744bf9f33ddc87fd59771: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:15.720142 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 14034
I20260929 19:03:15.720242 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:03:15.720321 14101 log.cc:824] T 00000000000000000000000000000000 P 2e45f7c5415744bf9f33ddc87fd59771: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:15.720453 8825 external_mini_cluster.cc:1428] Setting key 70db8cb7236d194139eb917a3e8a7602
I20260929 19:03:15.721148 14101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e45f7c5415744bf9f33ddc87fd59771: No bootstrap required, opened a new log
I20260929 19:03:15.723651 14101 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e45f7c5415744bf9f33ddc87fd59771 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e45f7c5415744bf9f33ddc87fd59771" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 44355 } }
I20260929 19:03:15.723811 14101 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e45f7c5415744bf9f33ddc87fd59771 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:15.723841 14101 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e45f7c5415744bf9f33ddc87fd59771 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e45f7c5415744bf9f33ddc87fd59771, State: Initialized, Role: FOLLOWER
I20260929 19:03:15.723974 14101 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e45f7c5415744bf9f33ddc87fd59771 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e45f7c5415744bf9f33ddc87fd59771" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 44355 } }
I20260929 19:03:15.724061 14101 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e45f7c5415744bf9f33ddc87fd59771 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:03:15.724099 14101 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e45f7c5415744bf9f33ddc87fd59771 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:03:15.724159 14101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e45f7c5415744bf9f33ddc87fd59771 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:15.724717 14101 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e45f7c5415744bf9f33ddc87fd59771 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e45f7c5415744bf9f33ddc87fd59771" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 44355 } }
I20260929 19:03:15.724836 14101 leader_election.cc:302] T 00000000000000000000000000000000 P 2e45f7c5415744bf9f33ddc87fd59771 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e45f7c5415744bf9f33ddc87fd59771; no voters:
I20260929 19:03:15.724993 14101 leader_election.cc:288] T 00000000000000000000000000000000 P 2e45f7c5415744bf9f33ddc87fd59771 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:03:15.725046 14106 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e45f7c5415744bf9f33ddc87fd59771 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:03:15.725181 14106 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e45f7c5415744bf9f33ddc87fd59771 [term 1 LEADER]: Becoming Leader. State: Replica: 2e45f7c5415744bf9f33ddc87fd59771, State: Running, Role: LEADER
I20260929 19:03:15.725272 14101 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e45f7c5415744bf9f33ddc87fd59771 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:03:15.725276 14106 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e45f7c5415744bf9f33ddc87fd59771 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e45f7c5415744bf9f33ddc87fd59771" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 44355 } }
I20260929 19:03:15.725629 14107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e45f7c5415744bf9f33ddc87fd59771 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e45f7c5415744bf9f33ddc87fd59771. Latest consensus state: current_term: 1 leader_uuid: "2e45f7c5415744bf9f33ddc87fd59771" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e45f7c5415744bf9f33ddc87fd59771" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 44355 } } }
I20260929 19:03:15.725696 14107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e45f7c5415744bf9f33ddc87fd59771 [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:15.725899 14112 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:03:15.726449 14105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.720971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34336 (local address 127.8.158.126:44355)
0929 19:03:15.721159 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:15.721171 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:15.721214 (+ 43us) server_negotiation.cc:407] Connection header received
0929 19:03:15.721648 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:15.721654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:15.721707 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:15.721815 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:15.722038 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.722973 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.724137 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.724633 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.724731 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:15.725438 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:15.725443 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:15.725443 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:03:15.725653 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:15.725695 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:03:15.725712 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:15.725775 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:15.726115 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 19:03:15.726196 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
I20260929 19:03:15.726526 14112 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:03:15.726751 14108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e45f7c5415744bf9f33ddc87fd59771 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e45f7c5415744bf9f33ddc87fd59771" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e45f7c5415744bf9f33ddc87fd59771" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 44355 } } }
I20260929 19:03:15.726954 14108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e45f7c5415744bf9f33ddc87fd59771 [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:15.727362 8825 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:03:15.728226 14112 catalog_manager.cc:1347] Generated new cluster ID: 741c3d5ed4f74024ae1a823d50948f39
I20260929 19:03:15.728281 14112 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:03:15.736995 14112 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:03:15.737474 14112 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:03:15.743237 14112 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e45f7c5415744bf9f33ddc87fd59771: Generated new TSK 0
I20260929 19:03:15.743400 14112 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 19:03:15.769738 14131 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 19:03:15.769834 14131 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 19:03:15.770073 14105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.756105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34346 (local address 127.8.158.126:44355)
0929 19:03:15.756150 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:15.756152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:15.756232 (+ 80us) server_negotiation.cc:407] Connection header received
0929 19:03:15.756270 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:15.756271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:15.756283 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:15.756383 (+ 100us) server_negotiation.cc:226] Negotiated authn=JWT
0929 19:03:15.756508 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.757283 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.757999 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.758303 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.758395 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:15.758548 (+ 153us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 19:03:15.769878 (+ 11330us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38699/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 19:03:15.770008 (+ 130us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34346: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38699/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":26}
W20260929 19:03:15.770124 14105 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34346: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38699/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260929 19:03:15.770151 14130 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:15.756024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:44355 (local address 127.0.0.1:34346)
0929 19:03:15.756155 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:15.756165 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:15.756227 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:15.756392 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:15.756392 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:15.756394 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0929 19:03:15.756441 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:15.756442 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.757359 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.757360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:15.757920 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:15.757922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.758403 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.758404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:15.758452 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:15.758457 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 19:03:15.769978 (+ 11521us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38699/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 19:03:15.770059 (+ 81us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.158.126:44355: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38699/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":88,"threads_started":1}
I20260929 19:03:15.770447 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 14034
2026-09-29T19:03:15Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (220 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 19:03:15.808326 8825 mini_oidc.cc:152] Starting JWKS server
I20260929 19:03:15.808954 8825 webserver.cc:460] Webserver started at https://127.0.0.1:39515/ using document root <none> and password file <none>
I20260929 19:03:15.808995 8825 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 19:03:15.809157 8825 webserver.cc:460] Webserver started at http://127.0.0.1:40245/ using document root <none> and password file <none>
2026-09-29T19:03:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:03:15Z Disabled control of system clock
I20260929 19:03:15.816740 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:38791
--webserver_interface=127.8.158.126
--webserver_port=0
--builtin_ntp_servers=127.8.158.84:44189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:38791
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39515/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 19:03:15.898561 14137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:15.898741 14137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:15.898761 14137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:15.899830 14137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:03:15.899865 14137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:15.899878 14137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:15.899890 14137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:03:15.899902 14137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 19:03:15.899915 14137 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 19:03:15.899926 14137 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39515/jwks.json
I20260929 19:03:15.901230 14137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:44189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:38791
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:38791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39515/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.158.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:15.901404 14137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:15.901616 14137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:15.906577 14143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:15.906672 14145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:15.906606 14142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:15.906674 14137 server_base.cc:1034] running on GCE node
I20260929 19:03:15.907027 14137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:15.907310 14137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:15.908483 14137 hybrid_clock.cc:648] HybridClock initialized: now 1790708595908472 us; error 53 us; skew 500 ppm
I20260929 19:03:15.909634 14137 webserver.cc:460] Webserver started at http://127.8.158.126:41405/ using document root <none> and password file <none>
I20260929 19:03:15.909884 14137 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:15.909961 14137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:15.910094 14137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:15.911181 14137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "932dfa18ffbf492f83d1356bfae3ff45"
format_stamp: "Formatted at 2026-09-29 19:03:15 on dist-test-slave-nmsz"
server_key: "d085ae2e2119414ec48291affdf32fc4"
server_key_iv: "4d1a46f8b39fa52b0ac9263777dd62a5"
server_key_version: "encryptionkey@0"
I20260929 19:03:15.911545 14137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "932dfa18ffbf492f83d1356bfae3ff45"
format_stamp: "Formatted at 2026-09-29 19:03:15 on dist-test-slave-nmsz"
server_key: "d085ae2e2119414ec48291affdf32fc4"
server_key_iv: "4d1a46f8b39fa52b0ac9263777dd62a5"
server_key_version: "encryptionkey@0"
I20260929 19:03:15.912803 14137 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:03:15.913527 14151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:15.913712 14137 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:03:15.913800 14137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "932dfa18ffbf492f83d1356bfae3ff45"
format_stamp: "Formatted at 2026-09-29 19:03:15 on dist-test-slave-nmsz"
server_key: "d085ae2e2119414ec48291affdf32fc4"
server_key_iv: "4d1a46f8b39fa52b0ac9263777dd62a5"
server_key_version: "encryptionkey@0"
I20260929 19:03:15.913887 14137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:15.921018 14137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:15.921273 14137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:15.921414 14137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:15.925173 14137 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.126:38791
I20260929 19:03:15.925200 14203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.126:38791 every 8 connection(s)
I20260929 19:03:15.925503 14137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:03:15.926113 14204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:15.928639 14204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 932dfa18ffbf492f83d1356bfae3ff45: Bootstrap starting.
I20260929 19:03:15.929127 14204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 932dfa18ffbf492f83d1356bfae3ff45: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:15.929343 14204 log.cc:824] T 00000000000000000000000000000000 P 932dfa18ffbf492f83d1356bfae3ff45: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:15.929920 14204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 932dfa18ffbf492f83d1356bfae3ff45: No bootstrap required, opened a new log
I20260929 19:03:15.930568 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 14137
I20260929 19:03:15.930665 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:03:15.930814 8825 external_mini_cluster.cc:1428] Setting key faaf84040b336b64eea8bb85d7d905ee
I20260929 19:03:15.931991 14204 raft_consensus.cc:348] T 00000000000000000000000000000000 P 932dfa18ffbf492f83d1356bfae3ff45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932dfa18ffbf492f83d1356bfae3ff45" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 38791 } }
I20260929 19:03:15.932150 14204 raft_consensus.cc:374] T 00000000000000000000000000000000 P 932dfa18ffbf492f83d1356bfae3ff45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:15.932189 14204 raft_consensus.cc:729] T 00000000000000000000000000000000 P 932dfa18ffbf492f83d1356bfae3ff45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 932dfa18ffbf492f83d1356bfae3ff45, State: Initialized, Role: FOLLOWER
I20260929 19:03:15.932317 14204 consensus_queue.cc:260] T 00000000000000000000000000000000 P 932dfa18ffbf492f83d1356bfae3ff45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932dfa18ffbf492f83d1356bfae3ff45" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 38791 } }
I20260929 19:03:15.932403 14204 raft_consensus.cc:388] T 00000000000000000000000000000000 P 932dfa18ffbf492f83d1356bfae3ff45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:03:15.932433 14204 raft_consensus.cc:482] T 00000000000000000000000000000000 P 932dfa18ffbf492f83d1356bfae3ff45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:03:15.932479 14204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 932dfa18ffbf492f83d1356bfae3ff45 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:15.932961 14204 raft_consensus.cc:504] T 00000000000000000000000000000000 P 932dfa18ffbf492f83d1356bfae3ff45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932dfa18ffbf492f83d1356bfae3ff45" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 38791 } }
I20260929 19:03:15.933046 14204 leader_election.cc:302] T 00000000000000000000000000000000 P 932dfa18ffbf492f83d1356bfae3ff45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 932dfa18ffbf492f83d1356bfae3ff45; no voters:
I20260929 19:03:15.933209 14204 leader_election.cc:288] T 00000000000000000000000000000000 P 932dfa18ffbf492f83d1356bfae3ff45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:03:15.933244 14209 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 932dfa18ffbf492f83d1356bfae3ff45 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:03:15.933378 14209 raft_consensus.cc:686] T 00000000000000000000000000000000 P 932dfa18ffbf492f83d1356bfae3ff45 [term 1 LEADER]: Becoming Leader. State: Replica: 932dfa18ffbf492f83d1356bfae3ff45, State: Running, Role: LEADER
I20260929 19:03:15.933445 14204 sys_catalog.cc:564] T 00000000000000000000000000000000 P 932dfa18ffbf492f83d1356bfae3ff45 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:03:15.933467 14209 consensus_queue.cc:237] T 00000000000000000000000000000000 P 932dfa18ffbf492f83d1356bfae3ff45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932dfa18ffbf492f83d1356bfae3ff45" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 38791 } }
I20260929 19:03:15.933744 14209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 932dfa18ffbf492f83d1356bfae3ff45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 932dfa18ffbf492f83d1356bfae3ff45. Latest consensus state: current_term: 1 leader_uuid: "932dfa18ffbf492f83d1356bfae3ff45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932dfa18ffbf492f83d1356bfae3ff45" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 38791 } } }
I20260929 19:03:15.933809 14209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 932dfa18ffbf492f83d1356bfae3ff45 [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:15.933754 14210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 932dfa18ffbf492f83d1356bfae3ff45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "932dfa18ffbf492f83d1356bfae3ff45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932dfa18ffbf492f83d1356bfae3ff45" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 38791 } } }
I20260929 19:03:15.933944 14210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 932dfa18ffbf492f83d1356bfae3ff45 [sys.catalog]: This master's current role is: LEADER
W20260929 19:03:15.935185 14224 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 932dfa18ffbf492f83d1356bfae3ff45: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 19:03:15.935246 14224 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:03:15.937017 14208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.931282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49906 (local address 127.8.158.126:38791)
0929 19:03:15.931457 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:15.931464 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:15.931487 (+ 23us) server_negotiation.cc:407] Connection header received
0929 19:03:15.931973 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:15.931979 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:15.932041 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:15.932167 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:15.932391 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.934002 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.935104 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.935570 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.935703 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:15.936333 (+ 630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:15.936336 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:15.936337 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:03:15.936508 (+ 171us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:15.936551 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:03:15.936562 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:15.936642 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:15.936714 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 19:03:15.936871 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":77,"threads_started":1}
I20260929 19:03:15.938026 8825 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:03:15.938382 14214 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:03:15.938572 14214 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:03:15.939865 14214 catalog_manager.cc:1347] Generated new cluster ID: 58998940a4474663a801522326ed7ace
I20260929 19:03:15.939917 14214 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:03:15.965734 14214 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:03:15.966439 14214 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:03:15.980742 14214 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 932dfa18ffbf492f83d1356bfae3ff45: Generated new TSK 0
I20260929 19:03:15.980919 14214 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:03:15.998236 14208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.994631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49920 (local address 127.8.158.126:38791)
0929 19:03:15.994719 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:15.994721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:15.994734 (+ 13us) server_negotiation.cc:407] Connection header received
0929 19:03:15.994824 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:15.994825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:15.994838 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:15.994926 (+ 88us) server_negotiation.cc:226] Negotiated authn=JWT
0929 19:03:15.995038 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.995773 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.996602 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.996886 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.996970 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:15.997138 (+ 168us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 19:03:15.998034 (+ 896us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39515/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 19:03:15.998165 (+ 131us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49920: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39515/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":67}
W20260929 19:03:15.998302 14208 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49920: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39515/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 19:03:16.000170 14233 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:15.994517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38791 (local address 127.0.0.1:49920)
0929 19:03:15.994683 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:15.994695 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:15.994752 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:15.994923 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:15.994924 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:15.994926 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0929 19:03:15.994970 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:15.994972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.995878 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.995879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:15.996507 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:15.996509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.996995 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.996996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:15.997038 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:15.997042 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 19:03:15.998138 (+ 1096us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39515/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 19:03:16.000074 (+ 1936us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.158.126:38791: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39515/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":1740,"thread_start_us":155,"threads_started":1}
I20260929 19:03:16.000515 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 14137
E20260929 19:03:16.016291 14234 webserver.cc:533] Webserver: sslize failed:
E20260929 19:03:16.016372 14234 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-29T19:03:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (230 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T19:03:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:03:16Z Disabled control of system clock
I20260929 19:03:16.046126 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:46085
--webserver_interface=127.8.158.126
--webserver_port=0
--builtin_ntp_servers=127.8.158.84:39229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:46085
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708573736807-8825-0/testchainca.cert with env {}
W20260929 19:03:16.126895 14238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:16.127153 14238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:16.127197 14238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:16.128536 14238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:03:16.128609 14238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:16.128635 14238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:16.128670 14238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:03:16.128697 14238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 19:03:16.128728 14238 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 19:03:16.128759 14238 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 19:03:16.130710 14238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:39229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:46085
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:46085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.158.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708573736807-8825-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:16.130982 14238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:16.131196 14238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:16.136206 14244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:16.136217 14246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:16.136348 14238 server_base.cc:1034] running on GCE node
W20260929 19:03:16.136229 14243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:16.136718 14238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:16.137019 14238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:16.138193 14238 hybrid_clock.cc:648] HybridClock initialized: now 1790708596138186 us; error 37 us; skew 500 ppm
I20260929 19:03:16.139639 14238 webserver.cc:460] Webserver started at http://127.8.158.126:38875/ using document root <none> and password file <none>
I20260929 19:03:16.139864 14238 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:16.139930 14238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:16.140064 14238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:16.141223 14238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "f593d850cde34def9b313d36e550edf0"
format_stamp: "Formatted at 2026-09-29 19:03:16 on dist-test-slave-nmsz"
server_key: "619a520431a5b2c8053deb4e655b104f"
server_key_iv: "64168faf118a0415893c21195ed5c93d"
server_key_version: "encryptionkey@0"
I20260929 19:03:16.141660 14238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "f593d850cde34def9b313d36e550edf0"
format_stamp: "Formatted at 2026-09-29 19:03:16 on dist-test-slave-nmsz"
server_key: "619a520431a5b2c8053deb4e655b104f"
server_key_iv: "64168faf118a0415893c21195ed5c93d"
server_key_version: "encryptionkey@0"
I20260929 19:03:16.142992 14238 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260929 19:03:16.143663 14252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:16.143829 14238 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:03:16.143927 14238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "f593d850cde34def9b313d36e550edf0"
format_stamp: "Formatted at 2026-09-29 19:03:16 on dist-test-slave-nmsz"
server_key: "619a520431a5b2c8053deb4e655b104f"
server_key_iv: "64168faf118a0415893c21195ed5c93d"
server_key_version: "encryptionkey@0"
I20260929 19:03:16.144002 14238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:16.160122 14238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:16.160373 14238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:16.160521 14238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:16.164219 14238 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.126:46085
I20260929 19:03:16.164252 14304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.126:46085 every 8 connection(s)
I20260929 19:03:16.164561 14238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:03:16.165439 14305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:16.167719 14305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f593d850cde34def9b313d36e550edf0: Bootstrap starting.
I20260929 19:03:16.168285 14305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f593d850cde34def9b313d36e550edf0: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:16.168571 14305 log.cc:824] T 00000000000000000000000000000000 P f593d850cde34def9b313d36e550edf0: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:16.169204 14305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f593d850cde34def9b313d36e550edf0: No bootstrap required, opened a new log
I20260929 19:03:16.170049 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 14238
I20260929 19:03:16.170128 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:03:16.170347 8825 external_mini_cluster.cc:1428] Setting key 4bb0782e1b8f98e22f17c1644f713a65
I20260929 19:03:16.171352 14305 raft_consensus.cc:348] T 00000000000000000000000000000000 P f593d850cde34def9b313d36e550edf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f593d850cde34def9b313d36e550edf0" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 46085 } }
I20260929 19:03:16.171517 14305 raft_consensus.cc:374] T 00000000000000000000000000000000 P f593d850cde34def9b313d36e550edf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:16.171555 14305 raft_consensus.cc:729] T 00000000000000000000000000000000 P f593d850cde34def9b313d36e550edf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f593d850cde34def9b313d36e550edf0, State: Initialized, Role: FOLLOWER
I20260929 19:03:16.171681 14305 consensus_queue.cc:260] T 00000000000000000000000000000000 P f593d850cde34def9b313d36e550edf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f593d850cde34def9b313d36e550edf0" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 46085 } }
I20260929 19:03:16.171761 14305 raft_consensus.cc:388] T 00000000000000000000000000000000 P f593d850cde34def9b313d36e550edf0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:03:16.171792 14305 raft_consensus.cc:482] T 00000000000000000000000000000000 P f593d850cde34def9b313d36e550edf0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:03:16.171844 14305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f593d850cde34def9b313d36e550edf0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:16.172557 14305 raft_consensus.cc:504] T 00000000000000000000000000000000 P f593d850cde34def9b313d36e550edf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f593d850cde34def9b313d36e550edf0" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 46085 } }
I20260929 19:03:16.172668 14305 leader_election.cc:302] T 00000000000000000000000000000000 P f593d850cde34def9b313d36e550edf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f593d850cde34def9b313d36e550edf0; no voters:
I20260929 19:03:16.172796 14305 leader_election.cc:288] T 00000000000000000000000000000000 P f593d850cde34def9b313d36e550edf0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:03:16.172952 14311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f593d850cde34def9b313d36e550edf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:03:16.173031 14305 sys_catalog.cc:564] T 00000000000000000000000000000000 P f593d850cde34def9b313d36e550edf0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:03:16.173094 14311 raft_consensus.cc:686] T 00000000000000000000000000000000 P f593d850cde34def9b313d36e550edf0 [term 1 LEADER]: Becoming Leader. State: Replica: f593d850cde34def9b313d36e550edf0, State: Running, Role: LEADER
I20260929 19:03:16.173204 14311 consensus_queue.cc:237] T 00000000000000000000000000000000 P f593d850cde34def9b313d36e550edf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f593d850cde34def9b313d36e550edf0" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 46085 } }
I20260929 19:03:16.173482 14311 sys_catalog.cc:455] T 00000000000000000000000000000000 P f593d850cde34def9b313d36e550edf0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f593d850cde34def9b313d36e550edf0. Latest consensus state: current_term: 1 leader_uuid: "f593d850cde34def9b313d36e550edf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f593d850cde34def9b313d36e550edf0" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 46085 } } }
I20260929 19:03:16.173610 14311 sys_catalog.cc:458] T 00000000000000000000000000000000 P f593d850cde34def9b313d36e550edf0 [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:16.174433 14312 sys_catalog.cc:455] T 00000000000000000000000000000000 P f593d850cde34def9b313d36e550edf0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f593d850cde34def9b313d36e550edf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f593d850cde34def9b313d36e550edf0" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 46085 } } }
I20260929 19:03:16.174522 14312 sys_catalog.cc:458] T 00000000000000000000000000000000 P f593d850cde34def9b313d36e550edf0 [sys.catalog]: This master's current role is: LEADER
W20260929 19:03:16.176258 14325 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f593d850cde34def9b313d36e550edf0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 19:03:16.176317 14325 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:03:16.176379 14326 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:03:16.176548 14326 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:03:16.176903 14309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:16.170815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33546 (local address 127.8.158.126:46085)
0929 19:03:16.171033 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:16.171041 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:16.171075 (+ 34us) server_negotiation.cc:407] Connection header received
0929 19:03:16.171313 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:16.171321 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:16.171397 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:16.171555 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:16.171747 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:16.173468 (+ 1721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:16.174495 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:16.175176 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:16.175328 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:16.176047 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:16.176051 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:16.176051 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:03:16.176267 (+ 216us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:16.176307 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:03:16.176322 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:16.176590 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:16.176624 (+ 34us) server_negotiation.cc:299] Negotiation successful
0929 19:03:16.176698 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":84,"threads_started":1}
I20260929 19:03:16.177742 8825 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:03:16.178390 14326 catalog_manager.cc:1347] Generated new cluster ID: eebabba8ceb040cf9fa6881384fd8376
I20260929 19:03:16.178442 14326 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:03:16.183024 14326 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:03:16.183517 14326 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:03:16.193176 14326 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f593d850cde34def9b313d36e550edf0: Generated new TSK 0
I20260929 19:03:16.193375 14326 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:03:16.350960 14309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:16.204835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33560 (local address 127.8.158.126:46085)
0929 19:03:16.204891 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:16.204893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:16.205022 (+ 129us) server_negotiation.cc:407] Connection header received
0929 19:03:16.205056 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:16.205057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:16.205069 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:16.205143 (+ 74us) server_negotiation.cc:226] Negotiated authn=JWT
0929 19:03:16.205263 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:16.206046 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:16.206925 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:16.207227 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:16.207297 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:16.207436 (+ 139us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 19:03:16.350742 (+143306us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 19:03:16.350898 (+ 156us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33560: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":35}
W20260929 19:03:16.351029 14309 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33560: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 19:03:16.351131 14334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:16.204756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:46085 (local address 127.0.0.1:33560)
0929 19:03:16.204941 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:16.204950 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:16.205003 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:16.205146 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:16.205147 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:16.205148 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0929 19:03:16.205195 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:16.205197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:16.206138 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:16.206139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:16.206838 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:16.206841 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:16.207306 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:16.207307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:16.207345 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:16.207349 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 19:03:16.350873 (+143524us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 19:03:16.350990 (+ 117us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.158.126:46085: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":82,"threads_started":1}
I20260929 19:03:16.351521 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 14238
2026-09-29T19:03:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (329 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (85 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (87 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:03:16 dist-test-slave-nmsz krb5kdc[14346](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:03:16 dist-test-slave-nmsz krb5kdc[14346](info): set up 2 sockets
Sep 29 19:03:16 dist-test-slave-nmsz krb5kdc[14346](info): commencing operation
krb5kdc: starting...
W20260929 19:03:17.550299 8825 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Sep 29 19:03:17 dist-test-slave-nmsz krb5kdc[14346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:03:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:03:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.158.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
I20260929 19:03:17.666692 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:34063
--webserver_interface=127.8.158.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:38681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:34063
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:17.745252 14359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:17.745450 14359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:17.745473 14359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:17.746654 14359 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:03:17.746701 14359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:17.746714 14359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:17.746728 14359 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:03:17.746740 14359 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:03:17.748250 14359 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:38681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:34063
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:34063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.158.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:17.748481 14359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:17.748723 14359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:17.753540 14364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:17.753554 14367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:17.753573 14359 server_base.cc:1034] running on GCE node
W20260929 19:03:17.753527 14365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:17.753978 14359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:17.754282 14359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:17.755482 14359 hybrid_clock.cc:648] HybridClock initialized: now 1790708597755461 us; error 42 us; skew 500 ppm
Sep 29 19:03:17 dist-test-slave-nmsz krb5kdc[14346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708597, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:17.757818 14359 init.cc:377] Logged in from keytab as kudu/127.8.158.126@KRBTEST.COM (short username kudu)
I20260929 19:03:17.758275 14359 webserver.cc:460] Webserver started at http://127.8.158.126:45717/ using document root <none> and password file <none>
I20260929 19:03:17.758520 14359 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:17.758591 14359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:17.758742 14359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:17.759774 14359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "27f73e4e124c4ea48bd78003d01a0416"
format_stamp: "Formatted at 2026-09-29 19:03:17 on dist-test-slave-nmsz"
server_key: "e10378a5ed737e8a09b1847e5959af28"
server_key_iv: "74610e93bc41e60897724610c8ab8ae0"
server_key_version: "encryptionkey@0"
I20260929 19:03:17.760130 14359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "27f73e4e124c4ea48bd78003d01a0416"
format_stamp: "Formatted at 2026-09-29 19:03:17 on dist-test-slave-nmsz"
server_key: "e10378a5ed737e8a09b1847e5959af28"
server_key_iv: "74610e93bc41e60897724610c8ab8ae0"
server_key_version: "encryptionkey@0"
I20260929 19:03:17.761408 14359 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260929 19:03:17.762111 14374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:17.762267 14359 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:03:17.762380 14359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "27f73e4e124c4ea48bd78003d01a0416"
format_stamp: "Formatted at 2026-09-29 19:03:17 on dist-test-slave-nmsz"
server_key: "e10378a5ed737e8a09b1847e5959af28"
server_key_iv: "74610e93bc41e60897724610c8ab8ae0"
server_key_version: "encryptionkey@0"
I20260929 19:03:17.762462 14359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:17.779493 14359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:17.781348 14359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:17.781497 14359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:17.785676 14359 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.126:34063
I20260929 19:03:17.785707 14426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.126:34063 every 8 connection(s)
I20260929 19:03:17.786037 14359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:03:17.786885 14427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:17.788987 14427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27f73e4e124c4ea48bd78003d01a0416: Bootstrap starting.
I20260929 19:03:17.789494 14427 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27f73e4e124c4ea48bd78003d01a0416: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:17.789737 14427 log.cc:824] T 00000000000000000000000000000000 P 27f73e4e124c4ea48bd78003d01a0416: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:17.790392 14427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27f73e4e124c4ea48bd78003d01a0416: No bootstrap required, opened a new log
I20260929 19:03:17.791111 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 14359
I20260929 19:03:17.791224 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:03:17.791414 8825 external_mini_cluster.cc:1428] Setting key cb29528fc75954a0239bae5473738502
I20260929 19:03:17.792296 14427 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27f73e4e124c4ea48bd78003d01a0416 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27f73e4e124c4ea48bd78003d01a0416" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 34063 } }
I20260929 19:03:17.792476 14427 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27f73e4e124c4ea48bd78003d01a0416 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:17.792514 14427 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27f73e4e124c4ea48bd78003d01a0416 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f73e4e124c4ea48bd78003d01a0416, State: Initialized, Role: FOLLOWER
I20260929 19:03:17.792681 14427 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27f73e4e124c4ea48bd78003d01a0416 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27f73e4e124c4ea48bd78003d01a0416" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 34063 } }
I20260929 19:03:17.792768 14427 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27f73e4e124c4ea48bd78003d01a0416 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:03:17.792809 14427 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27f73e4e124c4ea48bd78003d01a0416 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:03:17.792858 14427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27f73e4e124c4ea48bd78003d01a0416 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:17.793864 14427 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27f73e4e124c4ea48bd78003d01a0416 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27f73e4e124c4ea48bd78003d01a0416" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 34063 } }
I20260929 19:03:17.793968 14427 leader_election.cc:302] T 00000000000000000000000000000000 P 27f73e4e124c4ea48bd78003d01a0416 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27f73e4e124c4ea48bd78003d01a0416; no voters:
I20260929 19:03:17.794102 14427 leader_election.cc:288] T 00000000000000000000000000000000 P 27f73e4e124c4ea48bd78003d01a0416 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:03:17.794179 14432 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27f73e4e124c4ea48bd78003d01a0416 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:03:17.794365 14427 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27f73e4e124c4ea48bd78003d01a0416 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:03:17.794371 14432 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27f73e4e124c4ea48bd78003d01a0416 [term 1 LEADER]: Becoming Leader. State: Replica: 27f73e4e124c4ea48bd78003d01a0416, State: Running, Role: LEADER
I20260929 19:03:17.794581 14432 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27f73e4e124c4ea48bd78003d01a0416 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27f73e4e124c4ea48bd78003d01a0416" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 34063 } }
I20260929 19:03:17.795977 14433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27f73e4e124c4ea48bd78003d01a0416 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27f73e4e124c4ea48bd78003d01a0416" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27f73e4e124c4ea48bd78003d01a0416" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 34063 } } }
I20260929 19:03:17.795987 14434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27f73e4e124c4ea48bd78003d01a0416 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27f73e4e124c4ea48bd78003d01a0416. Latest consensus state: current_term: 1 leader_uuid: "27f73e4e124c4ea48bd78003d01a0416" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27f73e4e124c4ea48bd78003d01a0416" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 34063 } } }
I20260929 19:03:17.796667 14433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27f73e4e124c4ea48bd78003d01a0416 [sys.catalog]: This master's current role is: LEADER
W20260929 19:03:17.796156 14446 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 27f73e4e124c4ea48bd78003d01a0416: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 19:03:17.796730 14446 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:03:17.796790 14434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27f73e4e124c4ea48bd78003d01a0416 [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:17.796921 14449 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:03:17.798199 14449 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 29 19:03:17 dist-test-slave-nmsz krb5kdc[14346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:03:17.799762 14449 catalog_manager.cc:1347] Generated new cluster ID: 684eadeb550d4ba6b17f8fed797b70e4
I20260929 19:03:17.799830 14449 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:03:17.802735 14431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:17.791925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49372 (local address 127.8.158.126:34063)
0929 19:03:17.792127 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:17.792139 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:17.792171 (+ 32us) server_negotiation.cc:407] Connection header received
0929 19:03:17.792281 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:17.792289 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:17.792344 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:17.792496 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:17.793244 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:17.794666 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:17.795721 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:17.796357 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:17.796468 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:17.799138 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:17.799142 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:17.799142 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:17.799152 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:17.801137 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:17.801538 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:17.801541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:17.801541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:17.801660 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:17.801873 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:17.801874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:17.801874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:17.802134 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:17.802272 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:17.802441 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 19:03:17.802559 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":88,"threads_started":1}
I20260929 19:03:17.805227 14449 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:03:17.805711 14449 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:03:17.808440 14449 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27f73e4e124c4ea48bd78003d01a0416: Generated new TSK 0
I20260929 19:03:17.808601 14449 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.158.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab.
Entry for principal kudu/127.8.158.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab.
I20260929 19:03:17.840979 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.65:0
--local_ip_for_outbound_sockets=127.8.158.65
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:34063
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab
--principal=kudu/127.8.158.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:38681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:17.955641 14453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:17.955909 14453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:17.955940 14453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:17.957582 14453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:17.957633 14453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:17.957669 14453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.65
I20260929 19:03:17.959760 14453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:38681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab
--principal=kudu/127.8.158.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:34063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.65
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:17.959990 14453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:17.960243 14453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:17.965261 14461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:17.965272 14459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:17.965380 14458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:17.965754 14453 server_base.cc:1034] running on GCE node
I20260929 19:03:17.966027 14453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:17.966269 14453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:17.967538 14453 hybrid_clock.cc:648] HybridClock initialized: now 1790708597967520 us; error 34 us; skew 500 ppm
Sep 29 19:03:17 dist-test-slave-nmsz krb5kdc[14346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708597, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:17.970000 14453 init.cc:377] Logged in from keytab as kudu/127.8.158.65@KRBTEST.COM (short username kudu)
I20260929 19:03:17.970527 14453 webserver.cc:460] Webserver started at http://127.8.158.65:38967/ using document root <none> and password file <none>
I20260929 19:03:17.970745 14453 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:17.970813 14453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:17.970963 14453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:17.971995 14453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-0/data/instance:
uuid: "100e0d85ea6b4bb092d07eb7c4734910"
format_stamp: "Formatted at 2026-09-29 19:03:17 on dist-test-slave-nmsz"
server_key: "e56be7e6eb1e66a74446a8416652e51c"
server_key_iv: "577bff50ca170fdb4e1906f8f468de88"
server_key_version: "encryptionkey@0"
I20260929 19:03:17.972358 14453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance:
uuid: "100e0d85ea6b4bb092d07eb7c4734910"
format_stamp: "Formatted at 2026-09-29 19:03:17 on dist-test-slave-nmsz"
server_key: "e56be7e6eb1e66a74446a8416652e51c"
server_key_iv: "577bff50ca170fdb4e1906f8f468de88"
server_key_version: "encryptionkey@0"
I20260929 19:03:17.973701 14453 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:03:17.974557 14468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:17.974752 14453 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:03:17.974862 14453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-0/wal
uuid: "100e0d85ea6b4bb092d07eb7c4734910"
format_stamp: "Formatted at 2026-09-29 19:03:17 on dist-test-slave-nmsz"
server_key: "e56be7e6eb1e66a74446a8416652e51c"
server_key_iv: "577bff50ca170fdb4e1906f8f468de88"
server_key_version: "encryptionkey@0"
I20260929 19:03:17.974941 14453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:17.989557 14453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:17.991828 14453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:17.992019 14453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:17.992280 14453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:17.992610 14453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:17.992676 14453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:17.992722 14453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:17.992753 14453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:17.999411 14453 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.65:36699
I20260929 19:03:17.999449 14581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.65:36699 every 8 connection(s)
I20260929 19:03:17.999786 14453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
I20260929 19:03:18.006554 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 14453
I20260929 19:03:18.006656 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance
Sep 29 19:03:18 dist-test-slave-nmsz krb5kdc[14346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708597, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:03:18.006873 8825 external_mini_cluster.cc:1428] Setting key cf41cdccc1344c8d6e6c826b4c78cf36
I20260929 19:03:18.011492 14584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:18.000559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:34063 (local address 127.8.158.65:54047)
0929 19:03:18.000761 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:18.000806 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:18.001744 (+ 938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:18.001984 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:18.001987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:18.002406 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:18.002618 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:18.002628 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:18.003394 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:18.003395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:18.004247 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:18.004249 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:18.004713 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:18.004715 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:18.004852 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:18.005432 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:18.005440 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:18.007198 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:18.009821 (+ 2623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:18.009829 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:18.009831 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:18.010253 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:18.010557 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:18.010558 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:18.010558 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:18.010661 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:18.010978 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:18.010982 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:18.011091 (+ 109us) client_negotiation.cc:769] Sending connection context
0929 19:03:18.011175 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 19:03:18.011294 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":124,"threads_started":1}
I20260929 19:03:18.011518 14431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:18.000579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:54047 (local address 127.8.158.126:34063)
0929 19:03:18.000650 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:18.000652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:18.000894 (+ 242us) server_negotiation.cc:407] Connection header received
0929 19:03:18.001847 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:18.001849 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:18.001859 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:18.001927 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:18.002729 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:18.003278 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:18.004357 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:18.004606 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:18.004704 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:18.007288 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:18.007289 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:18.007289 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:18.007301 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:18.009720 (+ 2419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:18.010405 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:18.010407 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:18.010407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:18.010481 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:18.010737 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:18.010739 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:18.010739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:18.010907 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:18.010986 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:18.011228 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 19:03:18.011355 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 19:03:18.011976 14582 heartbeater.cc:347] Connected to a master server at 127.8.158.126:34063
I20260929 19:03:18.012089 14582 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:18.012326 14582 heartbeater.cc:511] Master 127.8.158.126:34063 requested a full tablet report, sending...
I20260929 19:03:18.013027 14391 ts_manager.cc:194] Registered new tserver with Master: 100e0d85ea6b4bb092d07eb7c4734910 (127.8.158.65:36699)
I20260929 19:03:18.013779 14391 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.65@KRBTEST.COM'} at 127.8.158.65:54047
WARNING: no policy specified for kudu/127.8.158.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab.
Entry for principal kudu/127.8.158.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab.
I20260929 19:03:18.037990 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.66:0
--local_ip_for_outbound_sockets=127.8.158.66
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:34063
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab
--principal=kudu/127.8.158.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:38681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:18.123522 14587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:18.123733 14587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:18.123756 14587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:18.125067 14587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:18.125124 14587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:18.125154 14587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.66
I20260929 19:03:18.126705 14587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:38681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab
--principal=kudu/127.8.158.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:34063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.66
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:18.126951 14587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:18.127194 14587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 19:03:18.132543 14587 server_base.cc:1034] running on GCE node
W20260929 19:03:18.132506 14595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:18.132424 14592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:18.132424 14593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:18.132992 14587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:18.133221 14587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:18.134397 14587 hybrid_clock.cc:648] HybridClock initialized: now 1790708598134381 us; error 28 us; skew 500 ppm
Sep 29 19:03:18 dist-test-slave-nmsz krb5kdc[14346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708598, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:18.137171 14587 init.cc:377] Logged in from keytab as kudu/127.8.158.66@KRBTEST.COM (short username kudu)
I20260929 19:03:18.137614 14587 webserver.cc:460] Webserver started at http://127.8.158.66:40213/ using document root <none> and password file <none>
I20260929 19:03:18.137790 14587 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:18.137835 14587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:18.137930 14587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:18.139027 14587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-1/data/instance:
uuid: "05acd29186ac45d28d0dbdc1d54d3fb8"
format_stamp: "Formatted at 2026-09-29 19:03:18 on dist-test-slave-nmsz"
server_key: "1df355fe58def093a8b7e019b216b08e"
server_key_iv: "3423c27261e5dfab20db38462168f24e"
server_key_version: "encryptionkey@0"
I20260929 19:03:18.139328 14587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance:
uuid: "05acd29186ac45d28d0dbdc1d54d3fb8"
format_stamp: "Formatted at 2026-09-29 19:03:18 on dist-test-slave-nmsz"
server_key: "1df355fe58def093a8b7e019b216b08e"
server_key_iv: "3423c27261e5dfab20db38462168f24e"
server_key_version: "encryptionkey@0"
I20260929 19:03:18.140605 14587 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:03:18.141360 14602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:18.141574 14587 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:18.141676 14587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-1/wal
uuid: "05acd29186ac45d28d0dbdc1d54d3fb8"
format_stamp: "Formatted at 2026-09-29 19:03:18 on dist-test-slave-nmsz"
server_key: "1df355fe58def093a8b7e019b216b08e"
server_key_iv: "3423c27261e5dfab20db38462168f24e"
server_key_version: "encryptionkey@0"
I20260929 19:03:18.141767 14587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:18.164484 14587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:18.166733 14587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:18.166898 14587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:18.167141 14587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:18.167451 14587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:18.167508 14587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:18.167550 14587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:18.167575 14587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:18.173355 14587 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.66:33433
I20260929 19:03:18.173393 14715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.66:33433 every 8 connection(s)
I20260929 19:03:18.173714 14587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
Sep 29 19:03:18 dist-test-slave-nmsz krb5kdc[14346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708598, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.66@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:03:18.182895 14431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:18.174474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:41313 (local address 127.8.158.126:34063)
0929 19:03:18.174516 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:18.174518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:18.174774 (+ 256us) server_negotiation.cc:407] Connection header received
0929 19:03:18.175732 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:18.175733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:18.175746 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:18.175803 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:18.176410 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:18.176927 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:18.177955 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:18.178165 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:18.178249 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:18.180329 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:18.180330 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:18.180331 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:18.180338 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:18.181520 (+ 1182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:18.181913 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:18.181914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:18.181914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:18.181977 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:18.182221 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:18.182222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:18.182222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:18.182384 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:18.182447 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:18.182712 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 19:03:18.182797 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 19:03:18.183068 14718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:18.174483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:34063 (local address 127.8.158.66:41313)
0929 19:03:18.174653 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:18.174689 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:18.175622 (+ 933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:18.175880 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:18.175883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:18.176126 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:18.176309 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:18.176313 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:18.177020 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:18.177021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:18.177867 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:18.177869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:18.178259 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:18.178260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:18.178432 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:18.178940 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:18.178948 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:18.180235 (+ 1287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:18.181616 (+ 1381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:18.181622 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:18.181624 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:18.181836 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:18.182042 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:18.182043 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:18.182043 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:18.182145 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:18.182445 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:18.182451 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:18.182581 (+ 130us) client_negotiation.cc:769] Sending connection context
0929 19:03:18.182693 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 19:03:18.182840 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":87,"threads_started":1}
I20260929 19:03:18.183336 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 14587
I20260929 19:03:18.183458 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance
I20260929 19:03:18.183545 14716 heartbeater.cc:347] Connected to a master server at 127.8.158.126:34063
I20260929 19:03:18.183638 14716 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:18.183655 8825 external_mini_cluster.cc:1428] Setting key 37d97fd472f4dab9829dca33983c9aa4
I20260929 19:03:18.183846 14716 heartbeater.cc:511] Master 127.8.158.126:34063 requested a full tablet report, sending...
I20260929 19:03:18.184265 14391 ts_manager.cc:194] Registered new tserver with Master: 05acd29186ac45d28d0dbdc1d54d3fb8 (127.8.158.66:33433)
I20260929 19:03:18.184638 14391 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.66@KRBTEST.COM'} at 127.8.158.66:41313
WARNING: no policy specified for kudu/127.8.158.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab.
Entry for principal kudu/127.8.158.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab.
I20260929 19:03:18.209769 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.67:0
--local_ip_for_outbound_sockets=127.8.158.67
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:34063
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab
--principal=kudu/127.8.158.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:38681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:18.290378 14721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:18.290578 14721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:18.290601 14721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:18.291759 14721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:18.291800 14721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:18.291826 14721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.67
I20260929 19:03:18.293462 14721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:38681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab
--principal=kudu/127.8.158.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:34063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.67
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:18.293687 14721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:18.293926 14721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:18.299291 14729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:18.299291 14726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:18.299292 14727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:18.299500 14721 server_base.cc:1034] running on GCE node
I20260929 19:03:18.299772 14721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:18.300015 14721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:18.301213 14721 hybrid_clock.cc:648] HybridClock initialized: now 1790708598301206 us; error 55 us; skew 500 ppm
Sep 29 19:03:18 dist-test-slave-nmsz krb5kdc[14346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708598, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:18.303772 14721 init.cc:377] Logged in from keytab as kudu/127.8.158.67@KRBTEST.COM (short username kudu)
I20260929 19:03:18.304241 14721 webserver.cc:460] Webserver started at http://127.8.158.67:45605/ using document root <none> and password file <none>
I20260929 19:03:18.304479 14721 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:18.304544 14721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:18.304692 14721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:18.305727 14721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-2/data/instance:
uuid: "85f3c97d76184d8087b03af9ed62aaab"
format_stamp: "Formatted at 2026-09-29 19:03:18 on dist-test-slave-nmsz"
server_key: "a4ee335df22666cb567d3169965f84c9"
server_key_iv: "6833fb53098fc8c964ba0cdf3566ce99"
server_key_version: "encryptionkey@0"
I20260929 19:03:18.306079 14721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance:
uuid: "85f3c97d76184d8087b03af9ed62aaab"
format_stamp: "Formatted at 2026-09-29 19:03:18 on dist-test-slave-nmsz"
server_key: "a4ee335df22666cb567d3169965f84c9"
server_key_iv: "6833fb53098fc8c964ba0cdf3566ce99"
server_key_version: "encryptionkey@0"
I20260929 19:03:18.307442 14721 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:03:18.308166 14736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:18.308440 14721 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:18.308533 14721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-2/wal
uuid: "85f3c97d76184d8087b03af9ed62aaab"
format_stamp: "Formatted at 2026-09-29 19:03:18 on dist-test-slave-nmsz"
server_key: "a4ee335df22666cb567d3169965f84c9"
server_key_iv: "6833fb53098fc8c964ba0cdf3566ce99"
server_key_version: "encryptionkey@0"
I20260929 19:03:18.308621 14721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:18.340221 14721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:18.342345 14721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:18.342504 14721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:18.342733 14721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:18.343030 14721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:18.343063 14721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:18.343086 14721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:18.343137 14721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:18.349092 14721 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.67:41131
I20260929 19:03:18.349169 14849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.67:41131 every 8 connection(s)
I20260929 19:03:18.349447 14721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
I20260929 19:03:18.354542 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 14721
I20260929 19:03:18.354635 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance
I20260929 19:03:18.354790 8825 external_mini_cluster.cc:1428] Setting key 8ec41977d80c4ce17c571b43bc75aee3
Sep 29 19:03:18 dist-test-slave-nmsz krb5kdc[14346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708598, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.67@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:03:18.358516 14431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:18.350191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:47507 (local address 127.8.158.126:34063)
0929 19:03:18.350233 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:18.350235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:18.350492 (+ 257us) server_negotiation.cc:407] Connection header received
0929 19:03:18.351406 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:18.351407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:18.351417 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:18.351478 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:18.352066 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:18.352561 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:18.353495 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:18.353681 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:18.353740 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:18.356119 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:18.356121 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:18.356121 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:18.356127 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:18.357345 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:18.357696 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:18.357697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:18.357698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:18.357748 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:18.357952 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:18.357953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:18.357953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:18.358059 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:18.358143 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:18.358386 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 19:03:18.358455 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 19:03:18.358736 14852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:18.350176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:34063 (local address 127.8.158.67:47507)
0929 19:03:18.350379 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:18.350422 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:18.351312 (+ 890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:18.351530 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:18.351533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:18.351785 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:18.351962 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:18.351966 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:18.352646 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:18.352646 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:18.353413 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:18.353415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:18.353757 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:18.353758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:18.353858 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:18.354393 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:18.354404 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:18.356019 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:18.357416 (+ 1397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:18.357421 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:18.357423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:18.357628 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:18.357808 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:18.357808 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:18.357809 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:18.357898 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:18.358111 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:18.358115 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:18.358217 (+ 102us) client_negotiation.cc:769] Sending connection context
0929 19:03:18.358366 (+ 149us) client_negotiation.cc:240] Negotiation successful
0929 19:03:18.358489 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":127,"threads_started":1}
I20260929 19:03:18.359192 14850 heartbeater.cc:347] Connected to a master server at 127.8.158.126:34063
I20260929 19:03:18.359268 14850 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:18.359398 14850 heartbeater.cc:511] Master 127.8.158.126:34063 requested a full tablet report, sending...
I20260929 19:03:18.359694 14391 ts_manager.cc:194] Registered new tserver with Master: 85f3c97d76184d8087b03af9ed62aaab (127.8.158.67:41131)
I20260929 19:03:18.360059 14391 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.67@KRBTEST.COM'} at 127.8.158.67:47507
I20260929 19:03:18.366593 8825 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/krb5kdc/test-admin.keytab.
Sep 29 19:03:18 dist-test-slave-nmsz krb5kdc[14346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:18.382754 8825 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 19:03:18.384539 14857 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:18.383641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54620 (local address 127.8.158.65:36699)
0929 19:03:18.383825 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:18.383830 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:18.383839 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:03:18.383862 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:18.383864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:18.383875 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:18.383941 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:18.384458 (+ 517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54620: BlockingRecv error: recv got EOF from 127.0.0.1:54620 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":111,"threads_started":1}
W20260929 19:03:18.384670 8825 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/krb5kdc/krb5cc' of type FILE
Sep 29 19:03:18 dist-test-slave-nmsz krb5kdc[14346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:18.385708 8825 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 19:03:18 dist-test-slave-nmsz krb5kdc[14346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:18.386586 8825 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 19:03:18 dist-test-slave-nmsz krb5kdc[14346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.158.65@KRBTEST.COM
I20260929 19:03:18.394522 14857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:18.387236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54634 (local address 127.8.158.65:36699)
0929 19:03:18.387296 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:18.387298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:18.387306 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:03:18.387370 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:18.387371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:18.387380 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:18.387430 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:18.387833 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:18.388551 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:18.389257 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:18.389565 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:18.389622 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:18.391839 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:18.391845 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:18.391845 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:18.391851 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:18.393338 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:18.393665 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:18.393668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:18.393668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:18.393717 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:18.393889 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:18.393890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:18.393890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:18.394001 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:18.394113 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:18.394247 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 19:03:18.394310 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260929 19:03:18.396481 14857 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:18.395984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54648 (local address 127.8.158.65:36699)
0929 19:03:18.396027 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:18.396029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:18.396061 (+ 32us) server_negotiation.cc:407] Connection header received
0929 19:03:18.396119 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:18.396120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:18.396128 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:18.396185 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:18.396437 (+ 252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54648: BlockingRecv error: recv got EOF from 127.0.0.1:54648 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260929 19:03:18.396571 8825 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/krb5kdc/krb5cc' of type FILE
Sep 29 19:03:18 dist-test-slave-nmsz krb5kdc[14346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:18.397505 8825 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 19:03:18 dist-test-slave-nmsz krb5kdc[14346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:18.398440 8825 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 19:03:18 dist-test-slave-nmsz krb5kdc[14346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.158.65@KRBTEST.COM
I20260929 19:03:18.405393 14857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:18.398910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54658 (local address 127.8.158.65:36699)
0929 19:03:18.398963 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:18.398965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:18.399028 (+ 63us) server_negotiation.cc:407] Connection header received
0929 19:03:18.399088 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:18.399089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:18.399098 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:18.399167 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:18.399514 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:18.400057 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:18.400729 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:18.400932 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:18.401015 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:18.403047 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:18.403049 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:18.403049 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:18.403054 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:18.404374 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:18.404687 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:18.404687 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:18.404688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:18.404738 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:18.404907 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:18.404907 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:18.404908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:18.405004 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:18.405077 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:18.405176 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 19:03:18.405234 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260929 19:03:18.406893 14857 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:18.406422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54664 (local address 127.8.158.65:36699)
0929 19:03:18.406474 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:18.406476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:18.406497 (+ 21us) server_negotiation.cc:407] Connection header received
0929 19:03:18.406538 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:18.406539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:18.406546 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:18.406598 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:18.406848 (+ 250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54664: BlockingRecv error: recv got EOF from 127.0.0.1:54664 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260929 19:03:18.406999 8825 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708573736807-8825-0/krb5kdc/krb5cc)
Sep 29 19:03:18 dist-test-slave-nmsz krb5kdc[14346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:18.407931 8825 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 19:03:18 dist-test-slave-nmsz krb5kdc[14346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:18.408846 8825 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 19:03:18 dist-test-slave-nmsz krb5kdc[14346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.158.65@KRBTEST.COM
I20260929 19:03:18.415555 14857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:18.409327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54670 (local address 127.8.158.65:36699)
0929 19:03:18.409374 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:18.409375 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:18.409427 (+ 52us) server_negotiation.cc:407] Connection header received
0929 19:03:18.409474 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:18.409475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:18.409481 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:18.409531 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:18.409904 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:18.410569 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:18.411361 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:18.411601 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:18.411671 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:18.413509 (+ 1838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:18.413510 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:18.413510 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:18.413516 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:18.414747 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:18.415037 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:18.415038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:18.415038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:18.415082 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:18.415237 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:18.415237 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:18.415237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:18.415313 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:18.415377 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:18.415456 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 19:03:18.415503 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 19:03:18.416162 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 14453
I20260929 19:03:18.421270 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 14587
I20260929 19:03:18.425277 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 14721
I20260929 19:03:18.429124 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 14359
2026-09-29T19:03:18Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1907 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:03:18 dist-test-slave-nmsz krb5kdc[14870](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:03:18 dist-test-slave-nmsz krb5kdc[14870](info): set up 2 sockets
Sep 29 19:03:18 dist-test-slave-nmsz krb5kdc[14870](info): commencing operation
krb5kdc: starting...
W20260929 19:03:19.458515 8825 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Sep 29 19:03:19 dist-test-slave-nmsz krb5kdc[14870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:03:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:03:19Z Disabled control of system clock
WARNING: no policy specified for oryx/127.8.158.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.158.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.158.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.126.keytab.
Entry for principal oryx/127.8.158.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.126.keytab.
I20260929 19:03:19.573052 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:37015
--webserver_interface=127.8.158.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.126.keytab
--principal=oryx/127.8.158.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:40011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:37015
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:19.650215 14883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:19.650415 14883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:19.650437 14883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:19.651553 14883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:03:19.651592 14883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:19.651607 14883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:19.651618 14883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:03:19.651630 14883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:03:19.653047 14883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:40011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:37015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.126.keytab
--principal=oryx/127.8.158.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:37015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.158.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:19.653262 14883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:19.653501 14883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:19.658046 14888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:19.658275 14891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:19.658272 14889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:19.658402 14883 server_base.cc:1034] running on GCE node
I20260929 19:03:19.658700 14883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:19.658963 14883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:19.660102 14883 hybrid_clock.cc:648] HybridClock initialized: now 1790708599660095 us; error 29 us; skew 500 ppm
Sep 29 19:03:19 dist-test-slave-nmsz krb5kdc[14870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708599, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.158.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:19.667011 14883 init.cc:377] Logged in from keytab as oryx/127.8.158.126@KRBTEST.COM (short username oryx)
I20260929 19:03:19.667475 14883 webserver.cc:460] Webserver started at http://127.8.158.126:43449/ using document root <none> and password file <none>
I20260929 19:03:19.667726 14883 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:19.667796 14883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:19.667946 14883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:19.669047 14883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "b3b7fb74082b41ab8c6bbdb28a50707f"
format_stamp: "Formatted at 2026-09-29 19:03:19 on dist-test-slave-nmsz"
server_key: "aa51a910e087097cd390eda5e43f7e70"
server_key_iv: "fc1a1a2662c6197e74338fef79cf1bca"
server_key_version: "encryptionkey@0"
I20260929 19:03:19.669430 14883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "b3b7fb74082b41ab8c6bbdb28a50707f"
format_stamp: "Formatted at 2026-09-29 19:03:19 on dist-test-slave-nmsz"
server_key: "aa51a910e087097cd390eda5e43f7e70"
server_key_iv: "fc1a1a2662c6197e74338fef79cf1bca"
server_key_version: "encryptionkey@0"
I20260929 19:03:19.670902 14883 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:03:19.671635 14898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:19.671803 14883 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:03:19.671906 14883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "b3b7fb74082b41ab8c6bbdb28a50707f"
format_stamp: "Formatted at 2026-09-29 19:03:19 on dist-test-slave-nmsz"
server_key: "aa51a910e087097cd390eda5e43f7e70"
server_key_iv: "fc1a1a2662c6197e74338fef79cf1bca"
server_key_version: "encryptionkey@0"
I20260929 19:03:19.671988 14883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:19.705171 14883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:19.707191 14883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:19.707355 14883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:19.711602 14883 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.126:37015
I20260929 19:03:19.711635 14950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.126:37015 every 8 connection(s)
I20260929 19:03:19.712010 14883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:03:19.712934 14951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:19.715147 14951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f: Bootstrap starting.
I20260929 19:03:19.715694 14951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:19.715971 14951 log.cc:824] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:19.716691 14951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f: No bootstrap required, opened a new log
I20260929 19:03:19.717360 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 14883
I20260929 19:03:19.717482 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:03:19.717689 8825 external_mini_cluster.cc:1428] Setting key 807b833acaad2356f9bac78fce15545a
I20260929 19:03:19.718730 14951 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } }
I20260929 19:03:19.718883 14951 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:19.718928 14951 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b7fb74082b41ab8c6bbdb28a50707f, State: Initialized, Role: FOLLOWER
I20260929 19:03:19.719043 14951 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } }
I20260929 19:03:19.719144 14951 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:03:19.719189 14951 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:03:19.719257 14951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:19.720095 14951 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } }
I20260929 19:03:19.720216 14951 leader_election.cc:302] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3b7fb74082b41ab8c6bbdb28a50707f; no voters:
I20260929 19:03:19.720415 14951 leader_election.cc:288] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:03:19.720446 14956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:03:19.720559 14956 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [term 1 LEADER]: Becoming Leader. State: Replica: b3b7fb74082b41ab8c6bbdb28a50707f, State: Running, Role: LEADER
I20260929 19:03:19.720639 14956 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } }
I20260929 19:03:19.720683 14951 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:03:19.720937 14956 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } }
I20260929 19:03:19.721017 14956 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:19.721138 14956 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3b7fb74082b41ab8c6bbdb28a50707f. Latest consensus state: current_term: 1 leader_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } }
I20260929 19:03:19.721210 14956 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [sys.catalog]: This master's current role is: LEADER
W20260929 19:03:19.722443 14972 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 19:03:19.722498 14972 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:03:19.722544 14961 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:03:19.722710 14961 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:03:19.724279 14961 catalog_manager.cc:1347] Generated new cluster ID: 7c8d5b9d44a245eba27c93910892b2b7
I20260929 19:03:19.724332 14961 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 19:03:19 dist-test-slave-nmsz krb5kdc[14870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM
I20260929 19:03:19.729223 14955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:19.718232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44584 (local address 127.8.158.126:37015)
0929 19:03:19.718426 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:19.718436 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:19.718468 (+ 32us) server_negotiation.cc:407] Connection header received
0929 19:03:19.718713 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:19.718719 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:19.718766 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:19.718914 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:19.719693 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:19.721115 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:19.722377 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:19.722955 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:19.723101 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:19.725764 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:19.725768 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:19.725768 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:19.725780 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:19.727632 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:19.728028 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:19.728031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:19.728032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:19.728148 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:19.728372 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:19.728374 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:19.728374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:19.728618 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:19.728809 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:19.728880 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 19:03:19.729024 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":102,"threads_started":1}
I20260929 19:03:19.738080 14961 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:03:19.738639 14961 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:03:19.740391 14961 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f: Generated new TSK 0
I20260929 19:03:19.740554 14961 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.8.158.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.158.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.158.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.65.keytab.
Entry for principal oryx/127.8.158.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.65.keytab.
I20260929 19:03:19.769459 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.65:0
--local_ip_for_outbound_sockets=127.8.158.65
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:37015
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.65.keytab
--principal=oryx/127.8.158.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:40011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:19.861464 14977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:19.861680 14977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:19.861706 14977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:19.862963 14977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:19.863008 14977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:19.863037 14977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.65
I20260929 19:03:19.864614 14977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:40011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.65.keytab
--principal=oryx/127.8.158.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:37015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.65
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:19.864858 14977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:19.865100 14977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:19.870371 14983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:19.870435 14982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:19.870385 14985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:19.870620 14977 server_base.cc:1034] running on GCE node
I20260929 19:03:19.870790 14977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:19.870980 14977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:19.872131 14977 hybrid_clock.cc:648] HybridClock initialized: now 1790708599872135 us; error 41 us; skew 500 ppm
Sep 29 19:03:19 dist-test-slave-nmsz krb5kdc[14870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708599, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.158.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:19.874686 14977 init.cc:377] Logged in from keytab as oryx/127.8.158.65@KRBTEST.COM (short username oryx)
I20260929 19:03:19.875121 14977 webserver.cc:460] Webserver started at http://127.8.158.65:33389/ using document root <none> and password file <none>
I20260929 19:03:19.875303 14977 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:19.875342 14977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:19.875442 14977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:19.876451 14977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-0/data/instance:
uuid: "cc3c76ee46724a5cb427c2f4ac12ab34"
format_stamp: "Formatted at 2026-09-29 19:03:19 on dist-test-slave-nmsz"
server_key: "cff48e66fc6275e410c4f67ae37d7b7c"
server_key_iv: "9433b3c2ef0ad4b1da257180ff627329"
server_key_version: "encryptionkey@0"
I20260929 19:03:19.876781 14977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance:
uuid: "cc3c76ee46724a5cb427c2f4ac12ab34"
format_stamp: "Formatted at 2026-09-29 19:03:19 on dist-test-slave-nmsz"
server_key: "cff48e66fc6275e410c4f67ae37d7b7c"
server_key_iv: "9433b3c2ef0ad4b1da257180ff627329"
server_key_version: "encryptionkey@0"
I20260929 19:03:19.878067 14977 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260929 19:03:19.878786 14992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:19.878964 14977 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:19.879058 14977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-0/wal
uuid: "cc3c76ee46724a5cb427c2f4ac12ab34"
format_stamp: "Formatted at 2026-09-29 19:03:19 on dist-test-slave-nmsz"
server_key: "cff48e66fc6275e410c4f67ae37d7b7c"
server_key_iv: "9433b3c2ef0ad4b1da257180ff627329"
server_key_version: "encryptionkey@0"
I20260929 19:03:19.879134 14977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:19.889111 14977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:19.891501 14977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:19.891664 14977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:19.891878 14977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:19.892184 14977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:19.892247 14977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:19.892292 14977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:19.892320 14977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:19.898443 14977 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.65:43095
I20260929 19:03:19.898489 15105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.65:43095 every 8 connection(s)
I20260929 19:03:19.898809 14977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
I20260929 19:03:19.904217 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 14977
I20260929 19:03:19.904333 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance
I20260929 19:03:19.904515 8825 external_mini_cluster.cc:1428] Setting key e5dea44cd6485fce3aeedc50c9575156
Sep 29 19:03:19 dist-test-slave-nmsz krb5kdc[14870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708599, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.158.65@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM
I20260929 19:03:19.908823 14955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:19.899473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:39215 (local address 127.8.158.126:37015)
0929 19:03:19.899515 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:19.899517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:19.899759 (+ 242us) server_negotiation.cc:407] Connection header received
0929 19:03:19.900302 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:19.900304 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:19.900320 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:19.900371 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:19.901341 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:19.901885 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:19.903092 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:19.903353 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:19.903431 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:19.905803 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:19.905805 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:19.905805 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:19.905813 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:19.907370 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:19.907898 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:19.907899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:19.907899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:19.907949 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:19.908228 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:19.908229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:19.908229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:19.908323 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:19.908373 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:19.908697 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 19:03:19.908762 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 19:03:19.909128 15108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:19.899459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:37015 (local address 127.8.158.65:39215)
0929 19:03:19.899639 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:19.899673 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:19.900183 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:19.900451 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:19.900456 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:19.901050 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:19.901235 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:19.901240 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:19.901984 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:19.901985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:19.902906 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:19.902908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:19.903450 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:19.903451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:19.903583 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:19.904075 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:19.904083 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:19.905699 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:19.907455 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:19.907460 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:19.907462 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:19.907814 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:19.908011 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:19.908012 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:19.908012 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:19.908160 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:19.908387 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:19.908392 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:19.908563 (+ 171us) client_negotiation.cc:769] Sending connection context
0929 19:03:19.908684 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 19:03:19.908867 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":90,"threads_started":1}
I20260929 19:03:19.909618 15106 heartbeater.cc:347] Connected to a master server at 127.8.158.126:37015
I20260929 19:03:19.909720 15106 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:19.909898 15106 heartbeater.cc:511] Master 127.8.158.126:37015 requested a full tablet report, sending...
I20260929 19:03:19.910493 14915 ts_manager.cc:194] Registered new tserver with Master: cc3c76ee46724a5cb427c2f4ac12ab34 (127.8.158.65:43095)
I20260929 19:03:19.911463 14915 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.158.65@KRBTEST.COM'} at 127.8.158.65:39215
WARNING: no policy specified for oryx/127.8.158.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.158.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.158.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.66.keytab.
Entry for principal oryx/127.8.158.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.66.keytab.
I20260929 19:03:19.933458 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.66:0
--local_ip_for_outbound_sockets=127.8.158.66
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:37015
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.66.keytab
--principal=oryx/127.8.158.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:40011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:20.015408 15111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:20.015601 15111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:20.015625 15111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:20.016680 15111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:20.016714 15111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:20.016739 15111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.66
I20260929 19:03:20.018076 15111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:40011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.66.keytab
--principal=oryx/127.8.158.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:37015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.66
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:20.018239 15111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:20.018482 15111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:20.023435 15116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:20.023443 15117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:20.023501 15119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:20.023680 15111 server_base.cc:1034] running on GCE node
I20260929 19:03:20.023900 15111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:20.024121 15111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:20.025269 15111 hybrid_clock.cc:648] HybridClock initialized: now 1790708600025260 us; error 29 us; skew 500 ppm
Sep 29 19:03:20 dist-test-slave-nmsz krb5kdc[14870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708600, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.158.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:20.027678 15111 init.cc:377] Logged in from keytab as oryx/127.8.158.66@KRBTEST.COM (short username oryx)
I20260929 19:03:20.028139 15111 webserver.cc:460] Webserver started at http://127.8.158.66:42209/ using document root <none> and password file <none>
I20260929 19:03:20.028353 15111 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:20.028417 15111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:20.028546 15111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:20.029568 15111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-1/data/instance:
uuid: "a7a17ab10f8e4693b7d4321931c44da6"
format_stamp: "Formatted at 2026-09-29 19:03:20 on dist-test-slave-nmsz"
server_key: "5a0f7b8e9cd40c1dadfb19e93f0ed72e"
server_key_iv: "cb71e25b73c956748beae0a1c0c6fcca"
server_key_version: "encryptionkey@0"
I20260929 19:03:20.029922 15111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance:
uuid: "a7a17ab10f8e4693b7d4321931c44da6"
format_stamp: "Formatted at 2026-09-29 19:03:20 on dist-test-slave-nmsz"
server_key: "5a0f7b8e9cd40c1dadfb19e93f0ed72e"
server_key_iv: "cb71e25b73c956748beae0a1c0c6fcca"
server_key_version: "encryptionkey@0"
I20260929 19:03:20.031297 15111 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:03:20.032002 15126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:20.032161 15111 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:20.032258 15111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-1/wal
uuid: "a7a17ab10f8e4693b7d4321931c44da6"
format_stamp: "Formatted at 2026-09-29 19:03:20 on dist-test-slave-nmsz"
server_key: "5a0f7b8e9cd40c1dadfb19e93f0ed72e"
server_key_iv: "cb71e25b73c956748beae0a1c0c6fcca"
server_key_version: "encryptionkey@0"
I20260929 19:03:20.032342 15111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:20.041968 15111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:20.043831 15111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:20.043984 15111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:20.044194 15111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:20.044473 15111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:20.044528 15111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:20.044565 15111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:20.044595 15111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:20.050508 15111 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.66:42523
I20260929 19:03:20.050559 15239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.66:42523 every 8 connection(s)
I20260929 19:03:20.050858 15111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
Sep 29 19:03:20 dist-test-slave-nmsz krb5kdc[14870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708600, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.158.66@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM
I20260929 19:03:20.057654 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 15111
I20260929 19:03:20.057758 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance
I20260929 19:03:20.057948 8825 external_mini_cluster.cc:1428] Setting key 702551a4b6fe263787d133c31524fd04
I20260929 19:03:20.059906 15242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:20.051499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:37015 (local address 127.8.158.66:35571)
0929 19:03:20.051676 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:20.051739 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:20.052214 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:20.052419 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:20.052422 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:20.052951 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:20.053133 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:20.053137 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:20.053789 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:20.053790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:20.054657 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:20.054659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:20.054995 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:20.054996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:20.055112 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:20.055535 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:20.055543 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:20.056791 (+ 1248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:20.058575 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:20.058597 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:20.058599 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:20.058846 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:20.059083 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:20.059084 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:20.059085 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:20.059187 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:20.059465 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:20.059468 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:20.059572 (+ 104us) client_negotiation.cc:769] Sending connection context
0929 19:03:20.059651 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 19:03:20.059760 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":91,"threads_started":1}
I20260929 19:03:20.059924 14955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:20.051548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:35571 (local address 127.8.158.126:37015)
0929 19:03:20.051627 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:20.051630 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:20.051839 (+ 209us) server_negotiation.cc:407] Connection header received
0929 19:03:20.052306 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:20.052307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:20.052318 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:20.052372 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:20.053227 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:20.053709 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:20.054732 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:20.054912 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:20.054964 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:20.056873 (+ 1909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:20.056875 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:20.056875 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:20.056881 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:20.058482 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:20.058917 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:20.058918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:20.058919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:20.058977 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:20.059262 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:20.059263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:20.059263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:20.059405 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:20.059476 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:20.059693 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 19:03:20.059783 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 19:03:20.060504 15240 heartbeater.cc:347] Connected to a master server at 127.8.158.126:37015
I20260929 19:03:20.060607 15240 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:20.060787 15240 heartbeater.cc:511] Master 127.8.158.126:37015 requested a full tablet report, sending...
I20260929 19:03:20.061118 14915 ts_manager.cc:194] Registered new tserver with Master: a7a17ab10f8e4693b7d4321931c44da6 (127.8.158.66:42523)
I20260929 19:03:20.061475 14915 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.158.66@KRBTEST.COM'} at 127.8.158.66:35571
WARNING: no policy specified for oryx/127.8.158.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.158.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.158.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.67.keytab.
Entry for principal oryx/127.8.158.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.67.keytab.
I20260929 19:03:20.084965 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.67:0
--local_ip_for_outbound_sockets=127.8.158.67
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:37015
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.67.keytab
--principal=oryx/127.8.158.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:40011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:20.162437 15245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:20.162667 15245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:20.162706 15245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:20.163825 15245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:20.163892 15245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:20.163936 15245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.67
I20260929 19:03:20.165344 15245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:40011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.67.keytab
--principal=oryx/127.8.158.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:37015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.67
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:20.165601 15245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:20.165824 15245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:20.170513 15253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:20.170521 15250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:20.170568 15251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:20.170740 15245 server_base.cc:1034] running on GCE node
I20260929 19:03:20.171061 15245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:20.171312 15245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:20.172473 15245 hybrid_clock.cc:648] HybridClock initialized: now 1790708600172460 us; error 45 us; skew 500 ppm
Sep 29 19:03:20 dist-test-slave-nmsz krb5kdc[14870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708600, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.158.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:20.175145 15245 init.cc:377] Logged in from keytab as oryx/127.8.158.67@KRBTEST.COM (short username oryx)
I20260929 19:03:20.175652 15245 webserver.cc:460] Webserver started at http://127.8.158.67:40249/ using document root <none> and password file <none>
I20260929 19:03:20.176000 15245 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:20.176075 15245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:20.176218 15245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:20.177266 15245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-2/data/instance:
uuid: "b57b14eb0e6a4cb5a5df9a09c73924e7"
format_stamp: "Formatted at 2026-09-29 19:03:20 on dist-test-slave-nmsz"
server_key: "e7063bc57b6bbeb11abe9a0dacf11347"
server_key_iv: "dd87072d70cf995320104be4b18cdc76"
server_key_version: "encryptionkey@0"
I20260929 19:03:20.177618 15245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance:
uuid: "b57b14eb0e6a4cb5a5df9a09c73924e7"
format_stamp: "Formatted at 2026-09-29 19:03:20 on dist-test-slave-nmsz"
server_key: "e7063bc57b6bbeb11abe9a0dacf11347"
server_key_iv: "dd87072d70cf995320104be4b18cdc76"
server_key_version: "encryptionkey@0"
I20260929 19:03:20.178999 15245 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:03:20.179731 15260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:20.179927 15245 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:03:20.180023 15245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-2/wal
uuid: "b57b14eb0e6a4cb5a5df9a09c73924e7"
format_stamp: "Formatted at 2026-09-29 19:03:20 on dist-test-slave-nmsz"
server_key: "e7063bc57b6bbeb11abe9a0dacf11347"
server_key_iv: "dd87072d70cf995320104be4b18cdc76"
server_key_version: "encryptionkey@0"
I20260929 19:03:20.180101 15245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:20.204154 15245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:20.206516 15245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:20.206681 15245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:20.206910 15245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:20.207206 15245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:20.207260 15245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:20.207302 15245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:20.207330 15245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:20.213042 15245 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.67:38403
I20260929 19:03:20.213080 15373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.67:38403 every 8 connection(s)
I20260929 19:03:20.213428 15245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
I20260929 19:03:20.219399 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 15245
I20260929 19:03:20.219517 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance
Sep 29 19:03:20 dist-test-slave-nmsz krb5kdc[14870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708600, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.158.67@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM
I20260929 19:03:20.219732 8825 external_mini_cluster.cc:1428] Setting key cd2c11ef5141949b3094b02786db396d
I20260929 19:03:20.223052 14955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:20.214145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:49645 (local address 127.8.158.126:37015)
0929 19:03:20.214220 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:20.214222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:20.214430 (+ 208us) server_negotiation.cc:407] Connection header received
0929 19:03:20.214899 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:20.214901 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:20.214910 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:20.214979 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:20.215826 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:20.216309 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:20.217507 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:20.217896 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:20.217963 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:20.220177 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:20.220178 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:20.220179 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:20.220185 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:20.221794 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:20.222275 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:20.222276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:20.222276 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:20.222374 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:20.222573 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:20.222573 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:20.222574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:20.222672 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:20.222717 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:20.222937 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 19:03:20.222992 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 19:03:20.223275 15376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:20.214112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:37015 (local address 127.8.158.67:49645)
0929 19:03:20.214292 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:20.214357 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:20.214812 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:20.215032 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:20.215035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:20.215560 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:20.215746 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:20.215750 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:20.216387 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:20.216387 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:20.217418 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:20.217421 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:20.217982 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:20.217983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:20.218080 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:20.218581 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:20.218589 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:20.220094 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:20.221869 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:20.221875 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:20.221877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:20.222206 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:20.222433 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:20.222434 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:20.222434 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:20.222525 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:20.222724 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:20.222728 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:20.222828 (+ 100us) client_negotiation.cc:769] Sending connection context
0929 19:03:20.222920 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 19:03:20.223028 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":112,"threads_started":1}
I20260929 19:03:20.223721 15374 heartbeater.cc:347] Connected to a master server at 127.8.158.126:37015
I20260929 19:03:20.223790 15374 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:20.223966 15374 heartbeater.cc:511] Master 127.8.158.126:37015 requested a full tablet report, sending...
I20260929 19:03:20.224311 14915 ts_manager.cc:194] Registered new tserver with Master: b57b14eb0e6a4cb5a5df9a09c73924e7 (127.8.158.67:38403)
I20260929 19:03:20.224673 14915 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.158.67@KRBTEST.COM'} at 127.8.158.67:49645
I20260929 19:03:20.231647 8825 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.8.158.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.158.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.158.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.125.keytab.
Entry for principal oryx/127.8.158.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.125.keytab.
I20260929 19:03:20.257447 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.125:42397
--webserver_interface=127.8.158.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.125.keytab
--principal=oryx/127.8.158.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:40011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:37015,127.8.158.125:42397
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:20.341531 15379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:20.341750 15379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:20.341785 15379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:20.342926 15379 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:03:20.342994 15379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:20.343021 15379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:20.343055 15379 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:03:20.343076 15379 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:03:20.344535 15379 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:40011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:37015,127.8.158.125:42397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.125.keytab
--principal=oryx/127.8.158.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.125:42397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.158.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
W20260929 19:03:20.344681 15379 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.8.158.126:37015,127.8.158.125:42397'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 19:03:20.344836 15379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:20.345046 15379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:20.349862 15387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:20.349862 15384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:20.349898 15385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:20.350050 15379 server_base.cc:1034] running on GCE node
I20260929 19:03:20.350431 15379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:20.350720 15379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:20.351890 15379 hybrid_clock.cc:648] HybridClock initialized: now 1790708600351867 us; error 42 us; skew 500 ppm
Sep 29 19:03:20 dist-test-slave-nmsz krb5kdc[14870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708600, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.158.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:20.355026 15379 init.cc:377] Logged in from keytab as oryx/127.8.158.125@KRBTEST.COM (short username oryx)
I20260929 19:03:20.355422 15379 webserver.cc:460] Webserver started at http://127.8.158.125:35005/ using document root <none> and password file <none>
I20260929 19:03:20.355648 15379 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:20.355715 15379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:20.355859 15379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:20.356832 15379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-1/data/instance:
uuid: "1f3c183506914e0eaa46bcb72ac9c400"
format_stamp: "Formatted at 2026-09-29 19:03:20 on dist-test-slave-nmsz"
server_key: "cb737c251003bb04d3aa7d883d6390f8"
server_key_iv: "de4c9f90c769d2067fc606211dc3dd52"
server_key_version: "encryptionkey@0"
I20260929 19:03:20.357173 15379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-1/wal/instance:
uuid: "1f3c183506914e0eaa46bcb72ac9c400"
format_stamp: "Formatted at 2026-09-29 19:03:20 on dist-test-slave-nmsz"
server_key: "cb737c251003bb04d3aa7d883d6390f8"
server_key_iv: "de4c9f90c769d2067fc606211dc3dd52"
server_key_version: "encryptionkey@0"
I20260929 19:03:20.358489 15379 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260929 19:03:20.359184 15394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:20.359356 15379 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:03:20.359448 15379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-1/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-1/wal
uuid: "1f3c183506914e0eaa46bcb72ac9c400"
format_stamp: "Formatted at 2026-09-29 19:03:20 on dist-test-slave-nmsz"
server_key: "cb737c251003bb04d3aa7d883d6390f8"
server_key_iv: "de4c9f90c769d2067fc606211dc3dd52"
server_key_version: "encryptionkey@0"
I20260929 19:03:20.359539 15379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:20.380604 15379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:20.382869 15379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:20.383018 15379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:20.386812 15379 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.125:42397
I20260929 19:03:20.386847 15446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.125:42397 every 8 connection(s)
I20260929 19:03:20.387178 15379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-1/data/info.pb
I20260929 19:03:20.388106 15447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:20.389511 15447 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:03:20.391850 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 15379
Sep 29 19:03:20 dist-test-slave-nmsz krb5kdc[14870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708600, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.158.125@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM
Sep 29 19:03:20 dist-test-slave-nmsz krb5kdc[14870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.158.125@KRBTEST.COM
I20260929 19:03:20.399495 14955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:20.390192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58572 (local address 127.8.158.126:37015)
0929 19:03:20.390262 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:20.390264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:20.390502 (+ 238us) server_negotiation.cc:407] Connection header received
0929 19:03:20.390889 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:20.390890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:20.390898 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:20.390953 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:20.391509 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:20.392056 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:20.393185 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:20.393371 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:20.393461 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:20.396172 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:20.396173 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:20.396173 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:20.396179 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:20.397547 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:20.398148 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:20.398150 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:20.398150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:20.398225 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:20.398655 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:20.398656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:20.398657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:20.398898 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:20.398993 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:20.399268 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 19:03:20.399376 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 19:03:20.399712 15449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:20.390178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:37015 (local address 127.0.0.1:58572)
0929 19:03:20.390370 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:20.390407 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:20.390813 (+ 406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:20.390998 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:20.391001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:20.391223 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:20.391395 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:20.391398 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:20.392201 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:20.392202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:20.393077 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:20.393079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:20.393482 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:20.393483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:20.393613 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:20.394291 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:20.394301 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:20.396048 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:20.397659 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:20.397665 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:20.397667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:20.398012 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:20.398401 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:20.398402 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:20.398402 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:20.398569 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:20.398983 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:20.398988 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:20.399100 (+ 112us) client_negotiation.cc:769] Sending connection context
0929 19:03:20.399215 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 19:03:20.399453 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":95,"threads_started":1}
I20260929 19:03:20.400290 15447 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:03:20.401712 15451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:20.392315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56856 (local address 127.8.158.125:42397)
0929 19:03:20.392587 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:20.392590 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:20.392604 (+ 14us) server_negotiation.cc:407] Connection header received
0929 19:03:20.392620 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:20.392622 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:20.392639 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:20.392723 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:20.393309 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:20.394234 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:20.395214 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:20.395499 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:20.395557 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:20.398157 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:20.398165 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:20.398165 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:20.398172 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:20.400449 (+ 2277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:20.400844 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:20.400846 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:20.400847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:20.400914 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:20.401164 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:20.401165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:20.401165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:20.401322 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:20.401487 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:20.401531 (+ 44us) server_negotiation.cc:299] Negotiation successful
0929 19:03:20.401607 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":88,"threads_started":1}
Sep 29 19:03:20 dist-test-slave-nmsz krb5kdc[14870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708600, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.158.125@KRBTEST.COM for oryx/127.8.158.125@KRBTEST.COM
I20260929 19:03:20.407985 15452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:20.400603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56868 (local address 127.8.158.125:42397)
0929 19:03:20.400741 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:20.400743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:20.400751 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:03:20.400770 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:20.400771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:20.400787 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:20.400883 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:20.401167 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:20.401960 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:20.402779 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:20.403004 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:20.403104 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:20.404758 (+ 1654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:20.404760 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:20.404760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:20.404765 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:20.406732 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:20.407142 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:20.407143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:20.407143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:20.407189 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:20.407391 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:20.407392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:20.407392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:20.407476 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:20.407536 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:20.407753 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 19:03:20.407865 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":118,"thread_start_us":94,"threads_started":1}
I20260929 19:03:20.407997 15449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:20.400543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.125:42397 (local address 127.0.0.1:56868)
0929 19:03:20.400583 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:20.400593 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:20.400629 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:20.400902 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:20.400902 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:20.401040 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:20.401093 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:20.401095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:20.402074 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:20.402075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:20.402679 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:20.402681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:20.403104 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:20.403104 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:20.403179 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:20.403486 (+ 307us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:20.403489 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:20.404694 (+ 1205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:20.406804 (+ 2110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:20.406824 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:20.406825 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:20.407063 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:20.407254 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:20.407255 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:20.407255 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:20.407329 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:20.407568 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:20.407569 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:20.407616 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 19:03:20.407700 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 19:03:20.407810 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260929 19:03:20.409804 15447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400: Bootstrap starting.
I20260929 19:03:20.410506 15447 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:20.410823 15447 log.cc:824] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:20.412286 15447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400: No bootstrap required, opened a new log
I20260929 19:03:20.413982 15447 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } }
I20260929 19:03:20.414096 15447 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:20.414120 15447 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f3c183506914e0eaa46bcb72ac9c400, State: Initialized, Role: FOLLOWER
I20260929 19:03:20.414268 15447 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } }
I20260929 19:03:20.414644 15447 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:03:20.414606 15454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } } }
I20260929 19:03:20.414731 15454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 19:03:20.416429 15465 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 19:03:20.416486 15465 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:03:20.418046 15379 master_runner.cc:428] Detected that this master 1f3c183506914e0eaa46bcb72ac9c400 is joining an existing cluster
I20260929 19:03:20.418109 15379 master_runner.cc:432] Initiating AddMaster RPC to add 127.8.158.125:42397
I20260929 19:03:20.426091 15452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:20.419138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56884 (local address 127.8.158.125:42397)
0929 19:03:20.419188 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:20.419189 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:20.419420 (+ 231us) server_negotiation.cc:407] Connection header received
0929 19:03:20.419506 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:20.419507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:20.419515 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:20.419540 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:20.419965 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:20.420549 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:20.421523 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:20.421774 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:20.421877 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:20.423000 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:20.423001 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:20.423002 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:20.423008 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:20.424432 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:20.425173 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:20.425174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:20.425174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:20.425227 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:20.425444 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:20.425445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:20.425458 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:20.425580 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:20.425669 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:20.425842 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 19:03:20.425984 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":19}
I20260929 19:03:20.426440 15472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:20.419232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.125:42397 (local address 127.0.0.1:56884)
0929 19:03:20.419357 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:20.419368 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:20.419441 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:20.419588 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:20.419588 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:20.419799 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:20.419879 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:20.419880 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:20.420646 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:20.420647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:20.421421 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:20.421423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:20.421895 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:20.421897 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:20.421967 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:20.422442 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:20.422448 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:20.422886 (+ 438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:20.424546 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:20.424546 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:20.424547 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:20.424783 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:20.425305 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:20.425306 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:20.425306 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:20.425379 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:20.425660 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:20.425660 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:20.425714 (+ 54us) client_negotiation.cc:769] Sending connection context
0929 19:03:20.425819 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 19:03:20.425929 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":59,"threads_started":1}
I20260929 19:03:20.427266 14955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:20.419094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58578 (local address 127.8.158.126:37015)
0929 19:03:20.419135 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:20.419137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:20.419591 (+ 454us) server_negotiation.cc:407] Connection header received
0929 19:03:20.419611 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:20.419612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:20.419620 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:20.419693 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:20.420201 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:20.420966 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:20.422119 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:20.422459 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:20.422536 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:20.424377 (+ 1841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:20.424378 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:20.424379 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:20.424387 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:20.426000 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:20.426569 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:20.426570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:20.426570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:20.426624 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:20.426810 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:20.426811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:20.426812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:20.426951 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:20.427010 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:20.427140 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 19:03:20.427205 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 19:03:20.427325 15471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:20.419051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:37015 (local address 127.0.0.1:58578)
0929 19:03:20.419345 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:20.419355 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:20.419433 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:20.419698 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:20.419699 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:20.419896 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:20.420112 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:20.420114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:20.421070 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:20.421071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:20.422005 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:20.422008 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:20.422542 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:20.422543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:20.422607 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:20.423239 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:20.423243 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:20.424278 (+ 1035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:20.426089 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:20.426089 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:20.426090 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:20.426478 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:20.426696 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:20.426697 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:20.426697 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:20.426754 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:20.427001 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:20.427002 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:20.427041 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 19:03:20.427123 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 19:03:20.427217 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":82,"threads_started":1}
Sep 29 19:03:20 dist-test-slave-nmsz krb5kdc[14870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708599, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.158.126@KRBTEST.COM for oryx/127.8.158.125@KRBTEST.COM
I20260929 19:03:20.437079 15452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:20.429620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56896 (local address 127.8.158.125:42397)
0929 19:03:20.429661 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:20.429664 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:20.429763 (+ 99us) server_negotiation.cc:407] Connection header received
0929 19:03:20.429857 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:20.429858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:20.429865 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:20.429907 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:20.430439 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:20.430856 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:20.431942 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:20.432111 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:20.432168 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:20.434244 (+ 2076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:20.434245 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:20.434245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:20.434252 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:20.435860 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:20.436236 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:20.436237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:20.436237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:20.436287 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:20.436511 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:20.436512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:20.436512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:20.436639 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:20.436712 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:20.436906 (+ 194us) server_negotiation.cc:299] Negotiation successful
0929 19:03:20.436977 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 19:03:20.437084 15473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:20.429543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.125:42397 (local address 127.0.0.1:56896)
0929 19:03:20.429677 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:20.429714 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:20.429786 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:20.429933 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:20.429936 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:20.430267 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:20.430347 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:20.430349 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:20.430938 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:20.430939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:20.431846 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:20.431848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:20.432184 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:20.432185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:20.432273 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:20.432668 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:20.432673 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:20.434154 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:20.435946 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:20.435953 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:20.435955 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:20.436153 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:20.436360 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:20.436361 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:20.436361 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:20.436419 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:20.436717 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:20.436723 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:20.436771 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 19:03:20.436866 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 19:03:20.436979 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":83,"threads_started":1}
I20260929 19:03:20.437660 14915 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.8.158.125:42397: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400"
member_type: NON_VOTER
last_known_addr {
host: "127.8.158.125"
port: 42397
}
attrs {
promote: true
}
}
dest_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f"
cas_config_opid_index: -1
I20260929 19:03:20.437992 14915 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: NON_VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: true } }
I20260929 19:03:20.438501 15474 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 1f3c183506914e0eaa46bcb72ac9c400 (127.8.158.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: NON_VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: true } } }
I20260929 19:03:20.438772 15421 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:20.511390 15421 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [term 1 FOLLOWER]: Refusing update from remote peer b3b7fb74082b41ab8c6bbdb28a50707f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 19:03:20.512749 15474 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.8.158.125:42397
I20260929 19:03:20.513016 15474 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: NON_VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:03:20.513763 15474 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: NON_VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: true } } }
I20260929 19:03:20.513883 15474 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:20.515712 15379 master.cc:537] Master@127.8.158.125:42397 shutting down...
I20260929 19:03:20.516032 15421 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 1f3c183506914e0eaa46bcb72ac9c400 (127.8.158.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: NON_VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: true } } }
I20260929 19:03:20.517506 15480 mvcc.cc:205] Tried to move back new op lower bound from 7334742424535105536 to 7334742424456105984. Current Snapshot: MvccSnapshot[applied={T|T < 7334742424467582976}]
WARNING: no policy specified for oryx/127.8.158.124@KRBTEST.COM; defaulting to no policy
I20260929 19:03:20.612440 15486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: NON_VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: true } } }
I20260929 19:03:20.612617 15486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [sys.catalog]: This master's current role is: LEARNER
W20260929 19:03:20.613030 15454 raft_consensus.cc:659] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400: failed to trigger leader election: Illegal state: only voting members can start elections: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: NON_VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: true } }
I20260929 19:03:20.613108 15482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: NON_VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: true } } }
I20260929 19:03:20.613210 15482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [sys.catalog]: This master's current role is: LEARNER
I20260929 19:03:20.613317 15479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3b7fb74082b41ab8c6bbdb28a50707f. Latest consensus state: current_term: 1 leader_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: NON_VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: true } } }
I20260929 19:03:20.613396 15479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [sys.catalog]: This master's current role is: LEARNER
I20260929 19:03:20.613602 15379 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [term 1 LEARNER]: Raft consensus shutting down.
I20260929 19:03:20.613736 15379 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [term 1 LEARNER]: Raft consensus is shut down!
I20260929 19:03:20.613783 15379 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400: stopping tablet replica
I20260929 19:03:20.656168 15379 master.cc:552] Master@127.8.158.125:42397 shutdown complete.
I20260929 19:03:20.656224 15379 master_runner.cc:305] Clearing existing system tablet
I20260929 19:03:20.656332 15379 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:20.656371 15379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:20.657104 15379 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:20.657648 15488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:20.657790 15379 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:20.657860 15379 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:20.657881 15379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-1/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-1/wal
uuid: "1f3c183506914e0eaa46bcb72ac9c400"
format_stamp: "Formatted at 2026-09-29 19:03:20 on dist-test-slave-nmsz"
server_key: "cb737c251003bb04d3aa7d883d6390f8"
server_key_iv: "de4c9f90c769d2067fc606211dc3dd52"
server_key_version: "encryptionkey@0"
I20260929 19:03:20.658978 15379 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:03:20.750113 15379 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 19:03:20.750202 15379 log.cc:1196] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 19:03:20.750453 15379 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400: Deleting consensus metadata
I20260929 19:03:20.752450 15379 master_runner.cc:315] Copying system tablet from 127.8.158.126:37015
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.158.124@KRBTEST.COM" created.
I20260929 19:03:20.753159 15379 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400: tablet copy: Beginning tablet copy session from remote peer at address 127.8.158.126:37015
I20260929 19:03:20.760087 15494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:20.753495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:37015 (local address 127.0.0.1:58582)
0929 19:03:20.753646 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:20.753659 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:20.753741 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:20.754661 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:20.754662 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:20.754811 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:20.754852 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:20.754853 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:20.755557 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:20.755558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:20.756138 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:20.756140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:20.756538 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:20.756538 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:20.756594 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:20.757154 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:20.757158 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:20.757445 (+ 287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:20.759045 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:20.759046 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:20.759047 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:20.759224 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:20.759443 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:20.759443 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:20.759444 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:20.759502 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:20.759757 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:20.759758 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:20.759801 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 19:03:20.759891 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 19:03:20.759992 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":100,"threads_started":1}
I20260929 19:03:20.760082 14955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:20.753595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58582 (local address 127.8.158.126:37015)
0929 19:03:20.753647 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:20.753649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:20.753741 (+ 92us) server_negotiation.cc:407] Connection header received
0929 19:03:20.753804 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:20.753805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:20.753814 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:20.753865 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:20.754941 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:20.755458 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:20.756254 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:20.756451 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:20.756536 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:20.757535 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:20.757537 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:20.757537 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:20.757544 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:20.758956 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:20.759312 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:20.759313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:20.759314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:20.759376 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:20.759573 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:20.759573 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:20.759574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:20.759688 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:20.759765 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:20.759916 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 19:03:20.759990 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 19:03:20.760517 14935 tablet_copy_service.cc:140] P b3b7fb74082b41ab8c6bbdb28a50707f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 1f3c183506914e0eaa46bcb72ac9c400 ({username='oryx', principal='oryx/127.8.158.125@KRBTEST.COM'} at 127.0.0.1:58582)
I20260929 19:03:20.760618 14935 tablet_copy_service.cc:161] P b3b7fb74082b41ab8c6bbdb28a50707f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 1f3c183506914e0eaa46bcb72ac9c400 at {username='oryx', principal='oryx/127.8.158.125@KRBTEST.COM'} at 127.0.0.1:58582: session id = 1f3c183506914e0eaa46bcb72ac9c400-00000000000000000000000000000000
I20260929 19:03:20.761240 14935 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 19:03:20.761763 15379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 19:03:20.762801 15379 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400: tablet copy: Starting download of 0 data blocks...
I20260929 19:03:20.762943 15379 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400: tablet copy: Starting download of 1 WAL segments...
I20260929 19:03:20.763821 15379 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 19:03:20.844833 14935 tablet_copy_service.cc:342] P b3b7fb74082b41ab8c6bbdb28a50707f: Request end of tablet copy session 1f3c183506914e0eaa46bcb72ac9c400-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.8.158.125@KRBTEST.COM'} at 127.0.0.1:58582
I20260929 19:03:20.844966 14935 tablet_copy_service.cc:433] P b3b7fb74082b41ab8c6bbdb28a50707f: ending tablet copy session 1f3c183506914e0eaa46bcb72ac9c400-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 1f3c183506914e0eaa46bcb72ac9c400
I20260929 19:03:20.847692 15379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:20.848975 15497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:20.848977 15496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:20.849066 15499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:20.849309 15379 server_base.cc:1034] running on GCE node
I20260929 19:03:20.849467 15379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:20.849622 15379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:20.850754 15379 hybrid_clock.cc:648] HybridClock initialized: now 1790708600850744 us; error 23 us; skew 500 ppm
I20260929 19:03:20.912369 15106 heartbeater.cc:503] Master 127.8.158.126:37015 was elected leader, sending a full tablet report...
I20260929 19:03:20.986646 15476 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: NON_VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: true } } }
I20260929 19:03:20.986801 15476 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:20.987129 15502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:20.986812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.125:42397 (local address 127.0.0.1:56906)
0929 19:03:20.987045 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:20.987092 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.125:42397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":85,"threads_started":1}
W20260929 19:03:20.987326 14902 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.158.125:42397: connect: Connection refused (error 111)
I20260929 19:03:20.987757 15502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:20.987656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.125:42397 (local address 127.0.0.1:56910)
0929 19:03:20.987707 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:20.987732 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.125:42397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 19:03:20.987881 14902 consensus_peers.cc:487] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f -> Peer 1f3c183506914e0eaa46bcb72ac9c400 (127.8.158.125:42397): Couldn't send request to peer 1f3c183506914e0eaa46bcb72ac9c400. Status: Network error: Client connection negotiation failed: client connection to 127.8.158.125:42397: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
Sep 29 19:03:20 dist-test-slave-nmsz krb5kdc[14870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708600, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.158.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:20.988521 15379 init.cc:377] Logged in from keytab as oryx/127.8.158.125@KRBTEST.COM (short username oryx)
I20260929 19:03:20.988845 15379 webserver.cc:460] Webserver started at http://127.8.158.125:33851/ using document root <none> and password file <none>
I20260929 19:03:20.989022 15379 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:20.989073 15379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:20.989648 15379 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:20.990374 15508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:20.990684 15379 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 19:03:20.990777 15379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-1/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-1/wal
uuid: "1f3c183506914e0eaa46bcb72ac9c400"
format_stamp: "Formatted at 2026-09-29 19:03:20 on dist-test-slave-nmsz"
server_key: "cb737c251003bb04d3aa7d883d6390f8"
server_key_iv: "de4c9f90c769d2067fc606211dc3dd52"
server_key_version: "encryptionkey@0"
I20260929 19:03:20.990863 15379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:21.028874 15379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:21.030277 15379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:21.032785 15379 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.125:42397
I20260929 19:03:21.034224 15560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.125:42397 every 8 connection(s)
I20260929 19:03:21.062244 15240 heartbeater.cc:503] Master 127.8.158.126:37015 was elected leader, sending a full tablet report...
I20260929 19:03:21.070067 15379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-1/data/info.pb
I20260929 19:03:21.071412 15561 sys_catalog.cc:263] Verifying existing consensus state
I20260929 19:03:21.109323 15561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400: Bootstrap starting.
I20260929 19:03:21.111969 15561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 19:03:21.112200 15561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400: Bootstrap complete.
I20260929 19:03:21.112416 15561 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: NON_VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: true } }
I20260929 19:03:21.112510 15561 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1f3c183506914e0eaa46bcb72ac9c400, State: Initialized, Role: LEARNER
I20260929 19:03:21.112542 15561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: NON_VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: true } }
I20260929 19:03:21.112712 15561 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:03:21.112705 15564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: NON_VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: true } } }
I20260929 19:03:21.112808 15564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [sys.catalog]: This master's current role is: LEARNER
I20260929 19:03:21.114116 15575 catalog_manager.cc:1259] Loaded cluster ID: 7c8d5b9d44a245eba27c93910892b2b7
I20260929 19:03:21.114193 15575 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400: loading cluster ID for follower catalog manager: success
I20260929 19:03:21.115041 15575 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400: acquiring CA information for follower catalog manager: success
I20260929 19:03:21.115301 15575 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 19:03:21.225387 15374 heartbeater.cc:503] Master 127.8.158.126:37015 was elected leader, sending a full tablet report...
I20260929 19:03:21.455201 15474 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: NON_VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: true } } }
I20260929 19:03:21.455350 15474 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:21.460108 15502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:21.455303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.125:42397 (local address 127.0.0.1:56914)
0929 19:03:21.455376 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:21.455389 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:21.455473 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:21.455729 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:21.455730 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:21.455733 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:21.455795 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:21.455796 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:21.456618 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:21.456619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:21.459270 (+ 2651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:21.459273 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:21.459856 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:21.459857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:21.459917 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:21.459946 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 19:03:21.460017 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 19:03:21.460054 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 19:03:21.460127 15576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:21.455395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56914 (local address 127.8.158.125:42397)
0929 19:03:21.455541 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:21.455544 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:21.455561 (+ 17us) server_negotiation.cc:407] Connection header received
0929 19:03:21.455585 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:21.455586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:21.455608 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:21.455689 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:21.455861 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:21.456547 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:21.459357 (+ 2810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:21.459750 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:21.459809 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:21.459859 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:21.460031 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 19:03:21.460065 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":73,"threads_started":1}
I20260929 19:03:21.460453 15564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3b7fb74082b41ab8c6bbdb28a50707f. Latest consensus state: current_term: 1 leader_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: NON_VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: true } } }
I20260929 19:03:21.460552 15564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [sys.catalog]: This master's current role is: LEARNER
I20260929 19:03:21.460757 15476 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f: attempting to promote NON_VOTER 1f3c183506914e0eaa46bcb72ac9c400 to VOTER
I20260929 19:03:21.460980 15476 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: false } }
I20260929 19:03:21.461380 15535 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [term 1 LEARNER]: Refusing update from remote peer b3b7fb74082b41ab8c6bbdb28a50707f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 19:03:21.461606 15476 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 19:03:21.462492 15474 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1f3c183506914e0eaa46bcb72ac9c400 (127.8.158.125) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: false } } }
I20260929 19:03:21.462687 15535 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1f3c183506914e0eaa46bcb72ac9c400 (127.8.158.125) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: false } } }
I20260929 19:03:21.556734 15474 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: false } } }
I20260929 19:03:21.556782 15564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: false } } }
I20260929 19:03:21.556869 15474 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:21.556869 15564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:03:21.556952 15474 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: false } } }
I20260929 19:03:21.557005 15474 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [sys.catalog]: This master's current role is: LEADER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.158.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.124.keytab.
Entry for principal oryx/127.8.158.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.124.keytab.
W20260929 19:03:21.689852 8825 mini_kdc.cc:262] Time spent creating service keytab for oryx/127.8.158.124: real 1.177s user 0.000s sys 0.003s
I20260929 19:03:21.690214 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.124:46133
--webserver_interface=127.8.158.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.124.keytab
--principal=oryx/127.8.158.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:40011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:37015,127.8.158.125:42397,127.8.158.124:46133
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:21.771432 15578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:21.771623 15578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:21.771646 15578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:21.772702 15578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:03:21.772737 15578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:21.772750 15578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:21.772763 15578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:03:21.772774 15578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:03:21.774169 15578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:40011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:37015,127.8.158.125:42397,127.8.158.124:46133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.124.keytab
--principal=oryx/127.8.158.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.124:46133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.158.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:21.774416 15578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:21.774605 15578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:21.779279 15584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:21.779296 15583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:21.779296 15586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:21.779552 15578 server_base.cc:1034] running on GCE node
I20260929 19:03:21.779801 15578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:21.780114 15578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:21.781288 15578 hybrid_clock.cc:648] HybridClock initialized: now 1790708601781263 us; error 34 us; skew 500 ppm
Sep 29 19:03:21 dist-test-slave-nmsz krb5kdc[14870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708601, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.158.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:21.784348 15578 init.cc:377] Logged in from keytab as oryx/127.8.158.124@KRBTEST.COM (short username oryx)
I20260929 19:03:21.784765 15578 webserver.cc:460] Webserver started at http://127.8.158.124:45731/ using document root <none> and password file <none>
I20260929 19:03:21.784986 15578 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:21.785030 15578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:21.785111 15578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:21.786046 15578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-2/data/instance:
uuid: "9cee7e7f18ff4e888a7b900cc2049cbf"
format_stamp: "Formatted at 2026-09-29 19:03:21 on dist-test-slave-nmsz"
server_key: "0e0cfb142280668e93d5a1c2073d9453"
server_key_iv: "096c4222247884c05fee363ad49f75e6"
server_key_version: "encryptionkey@0"
I20260929 19:03:21.786396 15578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-2/wal/instance:
uuid: "9cee7e7f18ff4e888a7b900cc2049cbf"
format_stamp: "Formatted at 2026-09-29 19:03:21 on dist-test-slave-nmsz"
server_key: "0e0cfb142280668e93d5a1c2073d9453"
server_key_iv: "096c4222247884c05fee363ad49f75e6"
server_key_version: "encryptionkey@0"
I20260929 19:03:21.787796 15578 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:03:21.788506 15593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:21.788679 15578 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:03:21.788756 15578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-2/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-2/wal
uuid: "9cee7e7f18ff4e888a7b900cc2049cbf"
format_stamp: "Formatted at 2026-09-29 19:03:21 on dist-test-slave-nmsz"
server_key: "0e0cfb142280668e93d5a1c2073d9453"
server_key_iv: "096c4222247884c05fee363ad49f75e6"
server_key_version: "encryptionkey@0"
I20260929 19:03:21.788807 15578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:21.805828 15578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:21.808090 15578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:21.808194 15578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:21.811934 15578 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.124:46133
I20260929 19:03:21.811965 15645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.124:46133 every 8 connection(s)
I20260929 19:03:21.812273 15578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-2/data/info.pb
I20260929 19:03:21.813216 15646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:21.814816 15646 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:03:21.815301 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 15578
Sep 29 19:03:21 dist-test-slave-nmsz krb5kdc[14870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708601, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.158.124@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM
Sep 29 19:03:21 dist-test-slave-nmsz krb5kdc[14870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.158.124@KRBTEST.COM
I20260929 19:03:21.824618 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:21.815943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58172 (local address 127.8.158.124:46133)
0929 19:03:21.816095 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:21.816098 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:21.816124 (+ 26us) server_negotiation.cc:407] Connection header received
0929 19:03:21.816329 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:21.816338 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:21.816357 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:21.816460 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:21.817060 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:21.817844 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:21.818618 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:21.818910 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:21.818997 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:21.821782 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:21.821790 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:21.821790 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:21.821803 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:21.823297 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:21.823646 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:21.823649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:21.823651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:21.823725 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:21.823928 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:21.823929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:21.823930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:21.824105 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:21.824254 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:21.824332 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 19:03:21.824441 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
I20260929 19:03:21.824626 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:21.815515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:37015 (local address 127.0.0.1:58584)
0929 19:03:21.815729 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:21.815773 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:21.816313 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:21.816565 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:21.816567 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:21.816810 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:21.816946 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:21.816950 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:21.817660 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:21.817661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:21.818428 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:21.818430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:21.818831 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:21.818832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:21.818942 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:21.819401 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:21.819408 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:21.821072 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:21.823155 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:21.823161 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:21.823163 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:21.823384 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:21.823611 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:21.823612 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:21.823612 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:21.823726 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:21.823993 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:21.823998 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:21.824108 (+ 110us) client_negotiation.cc:769] Sending connection context
0929 19:03:21.824202 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 19:03:21.824369 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":123,"threads_started":1}
I20260929 19:03:21.825148 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:21.815597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58584 (local address 127.8.158.126:37015)
0929 19:03:21.815726 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:21.815729 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:21.815863 (+ 134us) server_negotiation.cc:407] Connection header received
0929 19:03:21.816415 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:21.816417 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:21.816432 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:21.816541 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:21.817048 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:21.817559 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:21.818523 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:21.818720 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:21.818803 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:21.821159 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:21.821161 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:21.821161 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:21.821167 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:21.823050 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:21.823477 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:21.823478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:21.823479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:21.823537 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:21.823794 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:21.823795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:21.823796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:21.823913 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:21.823987 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:21.824230 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 19:03:21.824307 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":79,"threads_started":1}
I20260929 19:03:21.825598 15646 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 19:03:21 dist-test-slave-nmsz krb5kdc[14870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708601, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.158.124@KRBTEST.COM for oryx/127.8.158.125@KRBTEST.COM
I20260929 19:03:21.831768 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:21.825831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.125:42397 (local address 127.0.0.1:56922)
0929 19:03:21.825877 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:21.825887 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:21.825921 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:21.826062 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:21.826063 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:21.826161 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:21.826187 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:21.826188 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:21.826840 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:21.826841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:21.827307 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:21.827308 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:21.827644 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:21.827645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:21.827683 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:21.828087 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:21.828091 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:21.829142 (+ 1051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:21.830767 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:21.830768 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:21.830768 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:21.830944 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:21.831145 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:21.831146 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:21.831146 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:21.831201 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:21.831461 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:21.831462 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:21.831501 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 19:03:21.831588 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 19:03:21.831677 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 19:03:21.831775 15576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:21.825911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56922 (local address 127.8.158.125:42397)
0929 19:03:21.825968 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:21.825970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:21.825978 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:03:21.826000 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:21.826001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:21.826010 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:21.826061 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:21.826242 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:21.826755 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:21.827396 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:21.827580 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:21.827636 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:21.829234 (+ 1598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:21.829235 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:21.829235 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:21.829241 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:21.830674 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:21.831027 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:21.831028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:21.831029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:21.831082 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:21.831282 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:21.831283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:21.831283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:21.831391 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:21.831459 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:21.831611 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 19:03:21.831692 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 19:03:21.832199 15646 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46133 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 19:03:21 dist-test-slave-nmsz krb5kdc[14870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708601, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.158.124@KRBTEST.COM for oryx/127.8.158.124@KRBTEST.COM
I20260929 19:03:21.838686 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:21.832645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.124:46133 (local address 127.0.0.1:58182)
0929 19:03:21.832716 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:21.832731 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:21.832775 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:21.832910 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:21.832911 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:21.833075 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:21.833215 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:21.833218 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:21.833886 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:21.833887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:21.834438 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:21.834440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:21.834869 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:21.834870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:21.834926 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:21.835276 (+ 350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:21.835279 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:21.836488 (+ 1209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:21.837745 (+ 1257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:21.837746 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:21.837746 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:21.837904 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:21.838090 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:21.838090 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:21.838090 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:21.838139 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:21.838404 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:21.838405 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:21.838445 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 19:03:21.838518 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 19:03:21.838604 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 19:03:21.838742 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:21.832696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58182 (local address 127.8.158.124:46133)
0929 19:03:21.832741 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:21.832742 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:21.832775 (+ 33us) server_negotiation.cc:407] Connection header received
0929 19:03:21.832841 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:21.832842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:21.832850 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:21.832892 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:21.833323 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:21.833792 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:21.834516 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:21.834772 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:21.834828 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:21.836557 (+ 1729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:21.836558 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:21.836558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:21.836564 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:21.837678 (+ 1114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:21.837973 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:21.837973 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:21.837974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:21.838019 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:21.838190 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:21.838191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:21.838191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:21.838281 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:21.838395 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:21.838536 (+ 141us) server_negotiation.cc:299] Negotiation successful
0929 19:03:21.838649 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":20}
I20260929 19:03:21.840066 15646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf: Bootstrap starting.
I20260929 19:03:21.840857 15646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:21.841131 15646 log.cc:824] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:21.841828 15646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf: No bootstrap required, opened a new log
I20260929 19:03:21.843295 15646 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } } peers { permanent_uuid: "9cee7e7f18ff4e888a7b900cc2049cbf" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46133 } }
I20260929 19:03:21.843451 15646 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:21.843524 15646 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cee7e7f18ff4e888a7b900cc2049cbf, State: Initialized, Role: FOLLOWER
I20260929 19:03:21.843664 15646 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } } peers { permanent_uuid: "9cee7e7f18ff4e888a7b900cc2049cbf" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46133 } }
I20260929 19:03:21.843991 15646 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:03:21.843953 15653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } } peers { permanent_uuid: "9cee7e7f18ff4e888a7b900cc2049cbf" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46133 } } }
I20260929 19:03:21.844060 15653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf [sys.catalog]: This master's current role is: FOLLOWER
W20260929 19:03:21.845774 15664 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 19:03:21.845826 15664 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:03:21.848253 15578 master_runner.cc:428] Detected that this master 9cee7e7f18ff4e888a7b900cc2049cbf is joining an existing cluster
I20260929 19:03:21.848317 15578 master_runner.cc:432] Initiating AddMaster RPC to add 127.8.158.124:46133
I20260929 19:03:21.856254 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:21.849556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.124:46133 (local address 127.0.0.1:58190)
0929 19:03:21.850403 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:21.850413 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:21.850484 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:21.850618 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:21.850618 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:21.850786 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:21.850836 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:21.850838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:21.851477 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:21.851477 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:21.852002 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:21.852003 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:21.852486 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:21.852487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:21.852546 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:21.853029 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:21.853034 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:21.853416 (+ 382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:21.855286 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:21.855287 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:21.855287 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:21.855441 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:21.855673 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:21.855674 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:21.855674 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:21.855734 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:21.855998 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:21.855999 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:21.856051 (+ 52us) client_negotiation.cc:769] Sending connection context
0929 19:03:21.856106 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 19:03:21.856182 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"thread_start_us":85,"threads_started":1}
I20260929 19:03:21.856348 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:21.849501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58190 (local address 127.8.158.124:46133)
0929 19:03:21.849538 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:21.849540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:21.850454 (+ 914us) server_negotiation.cc:407] Connection header received
0929 19:03:21.850543 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:21.850543 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:21.850550 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:21.850574 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:21.850902 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:21.851415 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:21.852081 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:21.852366 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:21.852423 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:21.853523 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:21.853525 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:21.853526 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:21.853534 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:21.855205 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:21.855527 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:21.855528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:21.855528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:21.855587 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:21.855805 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:21.855806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:21.855806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:21.855923 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:21.856000 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:21.856150 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 19:03:21.856244 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":16}
I20260929 19:03:21.876076 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 14977
I20260929 19:03:21.878571 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:21.849525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58588 (local address 127.8.158.126:37015)
0929 19:03:21.849573 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:21.849575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:21.849761 (+ 186us) server_negotiation.cc:407] Connection header received
0929 19:03:21.849869 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:21.849870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:21.849882 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:21.849925 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:21.850252 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:21.851063 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:21.852149 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:21.852457 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:21.852499 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:21.875569 (+ 23070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:21.875571 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:21.875571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:21.875579 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:21.877325 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:21.877709 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:21.877710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:21.877711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:21.877762 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:21.877951 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:21.877951 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:21.877952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:21.878085 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:21.878146 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:21.878384 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 19:03:21.878474 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 19:03:21.878906 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:21.849471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:37015 (local address 127.0.0.1:58588)
0929 19:03:21.849681 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:21.849692 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:21.849790 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:21.849937 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:21.849938 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:21.850118 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:21.850171 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:21.850172 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:21.851157 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:21.851158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:21.852038 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:21.852041 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:21.852576 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:21.852576 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:21.852638 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:21.875021 (+ 22383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:21.875026 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:21.875447 (+ 421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:21.877421 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:21.877422 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:21.877422 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:21.877632 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:21.877825 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:21.877825 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:21.877826 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:21.877877 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:21.878171 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:21.878172 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:21.878227 (+ 55us) client_negotiation.cc:769] Sending connection context
0929 19:03:21.878290 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 19:03:21.878800 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":74,"threads_started":1}
I20260929 19:03:21.880195 15576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:21.849745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56936 (local address 127.8.158.125:42397)
0929 19:03:21.849785 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:21.849786 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:21.852175 (+ 2389us) server_negotiation.cc:407] Connection header received
0929 19:03:21.852257 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:21.852258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:21.852267 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:21.852296 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:21.852701 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:21.853400 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:21.854521 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:21.854703 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:21.854795 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:21.875468 (+ 20673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:21.875471 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:21.875471 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:21.875480 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:21.878248 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:21.879315 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:21.879317 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:21.879317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:21.879393 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:21.879590 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:21.879591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:21.879592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:21.879737 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:21.879789 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:21.880012 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 19:03:21.880104 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 19:03:21.880208 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:21.849744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.125:42397 (local address 127.0.0.1:56936)
0929 19:03:21.852127 (+ 2383us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:21.852135 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:21.852196 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:21.852349 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:21.852350 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:21.852544 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:21.852600 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:21.852601 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:21.853500 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:21.853500 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:21.854181 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:21.854183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:21.854797 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:21.854798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:21.854868 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:21.874874 (+ 20006us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:21.874881 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:21.875339 (+ 458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:21.878374 (+ 3035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:21.878375 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:21.878381 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:21.879226 (+ 845us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:21.879462 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:21.879463 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:21.879463 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:21.879527 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:21.879822 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:21.879823 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:21.879867 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 19:03:21.879929 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 19:03:21.880044 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2369,"thread_start_us":94,"threads_started":1}
I20260929 19:03:21.882692 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.65:43095
--local_ip_for_outbound_sockets=127.8.158.65
--tserver_master_addrs=127.8.158.126:37015
--webserver_port=33389
--webserver_interface=127.8.158.65
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.65.keytab
--principal=oryx/127.8.158.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:40011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.158.126:37015,127.8.158.125:42397
--tserver_master_addrs=127.8.158.126:37015,127.8.158.125:42397,127.8.158.124:46133 with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 19:03:21 dist-test-slave-nmsz krb5kdc[14870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708599, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.158.126@KRBTEST.COM for oryx/127.8.158.124@KRBTEST.COM
I20260929 19:03:21.891630 15502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:21.883268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.124:46133 (local address 127.0.0.1:58200)
0929 19:03:21.883346 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:21.883357 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:21.883392 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:21.883657 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:21.883658 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:21.883853 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:21.883928 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:21.883930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:21.884938 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:21.884939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:21.885795 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:21.885797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:21.886303 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:21.886305 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:21.886405 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:21.886862 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:21.886867 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:21.888522 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:21.890419 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:21.890420 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:21.890421 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:21.890653 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:21.890879 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:21.890881 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:21.890881 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:21.890952 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:21.891300 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:21.891301 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:21.891351 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 19:03:21.891427 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 19:03:21.891527 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 19:03:21.891743 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:21.883467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58200 (local address 127.8.158.124:46133)
0929 19:03:21.883509 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:21.883512 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:21.883518 (+ 6us) server_negotiation.cc:407] Connection header received
0929 19:03:21.883549 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:21.883550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:21.883559 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:21.884049 (+ 490us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:21.884093 (+ 44us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:21.884826 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:21.885900 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:21.886217 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:21.886286 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:21.888615 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:21.888616 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:21.888617 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:21.888623 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:21.890278 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:21.890737 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:21.890738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:21.890738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:21.890813 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:21.891087 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:21.891088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:21.891088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:21.891233 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:21.891318 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:21.891508 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 19:03:21.891616 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 19:03:21.892570 14915 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.8.158.124:46133: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9cee7e7f18ff4e888a7b900cc2049cbf"
member_type: NON_VOTER
last_known_addr {
host: "127.8.158.124"
port: 46133
}
attrs {
promote: true
}
}
dest_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f"
cas_config_opid_index: 6
I20260929 19:03:21.892779 14915 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: false } } peers { permanent_uuid: "9cee7e7f18ff4e888a7b900cc2049cbf" member_type: NON_VOTER last_known_addr { host: "127.8.158.124" port: 46133 } attrs { promote: true } }
I20260929 19:03:21.893347 15535 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [term 1 FOLLOWER]: Refusing update from remote peer b3b7fb74082b41ab8c6bbdb28a50707f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 19:03:21.893487 15620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:21.893559 15475 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 19:03:21.894196 15475 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9cee7e7f18ff4e888a7b900cc2049cbf (127.8.158.124) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: false } } peers { permanent_uuid: "9cee7e7f18ff4e888a7b900cc2049cbf" member_type: NON_VOTER last_known_addr { host: "127.8.158.124" port: 46133 } attrs { promote: true } } }
I20260929 19:03:21.895304 15535 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9cee7e7f18ff4e888a7b900cc2049cbf (127.8.158.124) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: false } } peers { permanent_uuid: "9cee7e7f18ff4e888a7b900cc2049cbf" member_type: NON_VOTER last_known_addr { host: "127.8.158.124" port: 46133 } attrs { promote: true } } }
W20260929 19:03:21.965659 15673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:21.965874 15673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:21.965911 15673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:21.967093 15673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:21.967159 15673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:21.967204 15673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.65
I20260929 19:03:21.968673 15673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:40011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.65.keytab
--principal=oryx/127.8.158.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.65:43095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.158.65
--webserver_port=33389
--tserver_master_addrs=127.8.158.126:37015,127.8.158.125:42397,127.8.158.124:46133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.65
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:21.968927 15673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:21.969141 15673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:21.973955 15681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:21.974023 15678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:21.974146 15679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:21.974421 15673 server_base.cc:1034] running on GCE node
I20260929 19:03:21.974643 15673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:21.974881 15673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:21.976038 15673 hybrid_clock.cc:648] HybridClock initialized: now 1790708601976026 us; error 29 us; skew 500 ppm
Sep 29 19:03:21 dist-test-slave-nmsz krb5kdc[14870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708601, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.158.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:21.978539 15673 init.cc:377] Logged in from keytab as oryx/127.8.158.65@KRBTEST.COM (short username oryx)
I20260929 19:03:21.978989 15673 webserver.cc:460] Webserver started at http://127.8.158.65:33389/ using document root <none> and password file <none>
I20260929 19:03:21.979218 15673 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:21.979270 15673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:21.980566 15673 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:21.981169 15688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:21.981356 15673 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 19:03:21.981436 15673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-0/wal
uuid: "cc3c76ee46724a5cb427c2f4ac12ab34"
format_stamp: "Formatted at 2026-09-29 19:03:19 on dist-test-slave-nmsz"
server_key: "cff48e66fc6275e410c4f67ae37d7b7c"
server_key_iv: "9433b3c2ef0ad4b1da257180ff627329"
server_key_version: "encryptionkey@0"
I20260929 19:03:21.981765 15673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:21.984779 15620 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf [term 1 FOLLOWER]: Refusing update from remote peer b3b7fb74082b41ab8c6bbdb28a50707f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 19:03:21.985114 15476 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cee7e7f18ff4e888a7b900cc2049cbf" member_type: NON_VOTER last_known_addr { host: "127.8.158.124" port: 46133 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:03:21.986450 15620 raft_consensus.cc:858] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } } peers { permanent_uuid: "9cee7e7f18ff4e888a7b900cc2049cbf" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46133 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: NON_VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: false } }
W20260929 19:03:21.986498 15620 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } } peers { permanent_uuid: "9cee7e7f18ff4e888a7b900cc2049cbf" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46133 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: NON_VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: false } }
I20260929 19:03:21.986910 15620 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 1f3c183506914e0eaa46bcb72ac9c400 (127.8.158.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: NON_VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: true } } }
I20260929 19:03:21.988351 15653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3b7fb74082b41ab8c6bbdb28a50707f. Latest consensus state: current_term: 1 leader_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: NON_VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: true } } }
I20260929 19:03:21.988413 15653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260929 19:03:21.988440 15653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: NON_VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: true } } }
I20260929 19:03:21.988467 15653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260929 19:03:21.988507 15695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: NON_VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: true } } }
I20260929 19:03:21.988538 15695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260929 19:03:21.988901 15475 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.8.158.124:46133
I20260929 19:03:21.988988 15476 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: false } } peers { permanent_uuid: "9cee7e7f18ff4e888a7b900cc2049cbf" member_type: NON_VOTER last_known_addr { host: "127.8.158.124" port: 46133 } attrs { promote: true } } }
I20260929 19:03:21.989078 15476 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:21.989118 15476 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: false } } peers { permanent_uuid: "9cee7e7f18ff4e888a7b900cc2049cbf" member_type: NON_VOTER last_known_addr { host: "127.8.158.124" port: 46133 } attrs { promote: true } } }
I20260929 19:03:21.989152 15476 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:21.989312 15620 raft_consensus.cc:1467] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf [term 1 NON_PARTICIPANT]: Allowing update even though not a member of the config
I20260929 19:03:21.989403 15620 raft_consensus.cc:858] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: NON_VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: true } }.
Pending config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: false } }:
New config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: false } } peers { permanent_uuid: "9cee7e7f18ff4e888a7b900cc2049cbf" member_type: NON_VOTER last_known_addr { host: "127.8.158.124" port: 46133 } attrs { promote: true } }
W20260929 19:03:21.989420 15620 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 7' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: NON_VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: true } }.
Pending config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: false } }:
New config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: false } } peers { permanent_uuid: "9cee7e7f18ff4e888a7b900cc2049cbf" member_type: NON_VOTER last_known_addr { host: "127.8.158.124" port: 46133 } attrs { promote: true } }
I20260929 19:03:21.989461 15620 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1f3c183506914e0eaa46bcb72ac9c400 (127.8.158.125) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: false } } }
I20260929 19:03:21.989516 15564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: false } } peers { permanent_uuid: "9cee7e7f18ff4e888a7b900cc2049cbf" member_type: NON_VOTER last_known_addr { host: "127.8.158.124" port: 46133 } attrs { promote: true } } }
I20260929 19:03:21.989619 15564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:03:21.989887 15578 master.cc:537] Master@127.8.158.124:46133 shutting down...
I20260929 19:03:21.989004 15475 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: false } } peers { permanent_uuid: "9cee7e7f18ff4e888a7b900cc2049cbf" member_type: NON_VOTER last_known_addr { host: "127.8.158.124" port: 46133 } attrs { promote: true } } }
I20260929 19:03:21.990144 15475 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:21.990552 15693 mvcc.cc:205] Tried to move back new op lower bound from 7334742424535105536 to 7334742424456105984. Current Snapshot: MvccSnapshot[applied={T|T < 7334742424467582976}]
I20260929 19:03:22.012382 15673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:22.014745 15673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:22.014900 15673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:22.015163 15673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:22.015472 15673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:22.015527 15673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:22.015566 15673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:22.015594 15673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:22.021433 15673 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.65:43095
I20260929 19:03:22.021467 15808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.65:43095 every 8 connection(s)
I20260929 19:03:22.021836 15673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
Sep 29 19:03:22 dist-test-slave-nmsz krb5kdc[14870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708601, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.158.65@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM
Sep 29 19:03:22 dist-test-slave-nmsz krb5kdc[14870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708601, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.158.65@KRBTEST.COM for oryx/127.8.158.124@KRBTEST.COM
Sep 29 19:03:22 dist-test-slave-nmsz krb5kdc[14870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708601, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.158.65@KRBTEST.COM for oryx/127.8.158.125@KRBTEST.COM
I20260929 19:03:22.029908 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 15673
I20260929 19:03:22.030076 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 15111
I20260929 19:03:22.033631 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:22.022857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:50961 (local address 127.8.158.126:37015)
0929 19:03:22.022899 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:22.022901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:22.023083 (+ 182us) server_negotiation.cc:407] Connection header received
0929 19:03:22.023449 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:22.023450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:22.023462 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:22.023517 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:22.024360 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.024843 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.025818 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.026076 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.026141 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:22.028488 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:22.028489 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:22.028490 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:22.028497 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:22.031901 (+ 3404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:22.032455 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:22.032457 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:22.032458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:22.032520 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:22.032824 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:22.032825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:22.032826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:22.032975 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:22.033042 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:22.033434 (+ 392us) server_negotiation.cc:299] Negotiation successful
0929 19:03:22.033528 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 19:03:22.033690 15576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:22.022915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:43451 (local address 127.8.158.125:42397)
0929 19:03:22.022964 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:22.022966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:22.022976 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:03:22.023473 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:22.023474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:22.023487 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:22.023519 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:22.024908 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.025655 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.026612 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.026907 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.026979 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:22.029531 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:22.029532 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:22.029533 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:22.029540 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:22.032350 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:22.032738 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:22.032739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:22.032740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:22.032809 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:22.033025 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:22.033027 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:22.033027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:22.033168 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:22.033219 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:22.033451 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 19:03:22.033547 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 19:03:22.034162 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:22.022555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:37015 (local address 127.8.158.65:50961)
0929 19:03:22.023003 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:22.023012 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:22.023363 (+ 351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:22.023858 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:22.023858 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:22.024079 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:22.024260 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:22.024264 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.024946 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.024946 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:22.025728 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:22.025730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.026160 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.026160 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:22.026286 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:22.026774 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:22.026785 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:22.028377 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:22.031994 (+ 3617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:22.032000 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:22.032002 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:22.032357 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:22.032588 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:22.032589 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:22.032589 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:22.032718 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:22.033074 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:22.033080 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:22.033243 (+ 163us) client_negotiation.cc:769] Sending connection context
0929 19:03:22.033734 (+ 491us) client_negotiation.cc:240] Negotiation successful
0929 19:03:22.033915 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":26,"spinlock_wait_cycles":9472,"thread_start_us":95,"threads_started":1}
I20260929 19:03:22.034446 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:22.022553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.125:42397 (local address 127.8.158.65:43451)
0929 19:03:22.022697 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:22.022737 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:22.023352 (+ 615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:22.023622 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:22.023622 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:22.024759 (+ 1137us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:22.024814 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:22.024815 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.025740 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.025741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:22.026536 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:22.026539 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.026978 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.026978 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:22.027046 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:22.027437 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:22.027442 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:22.029445 (+ 2003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:22.032454 (+ 3009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:22.032455 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:22.032455 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:22.032641 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:22.032888 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:22.032888 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:22.032889 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:22.032948 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:22.033261 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:22.033262 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:22.033313 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 19:03:22.033971 (+ 658us) client_negotiation.cc:240] Negotiation successful
0929 19:03:22.034091 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":75,"threads_started":1}
I20260929 19:03:22.035427 15810 heartbeater.cc:347] Connected to a master server at 127.8.158.126:37015
I20260929 19:03:22.035516 15810 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:22.035676 15811 heartbeater.cc:347] Connected to a master server at 127.8.158.125:42397
I20260929 19:03:22.035701 15810 heartbeater.cc:511] Master 127.8.158.126:37015 requested a full tablet report, sending...
I20260929 19:03:22.035759 15811 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:22.035835 15811 heartbeater.cc:511] Master 127.8.158.125:42397 requested a full tablet report, sending...
I20260929 19:03:22.036154 14915 ts_manager.cc:194] Re-registered known tserver with Master: cc3c76ee46724a5cb427c2f4ac12ab34 (127.8.158.65:43095)
I20260929 19:03:22.036322 15525 ts_manager.cc:194] Registered new tserver with Master: cc3c76ee46724a5cb427c2f4ac12ab34 (127.8.158.65:43095)
I20260929 19:03:22.036675 14915 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.158.65@KRBTEST.COM'} at 127.8.158.65:50961
I20260929 19:03:22.037683 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.66:42523
--local_ip_for_outbound_sockets=127.8.158.66
--tserver_master_addrs=127.8.158.126:37015
--webserver_port=42209
--webserver_interface=127.8.158.66
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.66.keytab
--principal=oryx/127.8.158.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:40011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.158.126:37015,127.8.158.125:42397
--tserver_master_addrs=127.8.158.126:37015,127.8.158.125:42397,127.8.158.124:46133 with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:03:22.038617 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:22.022789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:46625 (local address 127.8.158.124:46133)
0929 19:03:22.022836 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:22.022839 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:22.022850 (+ 11us) server_negotiation.cc:407] Connection header received
0929 19:03:22.023473 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:22.023474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:22.023486 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:22.023519 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:22.024364 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.025031 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.025985 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.026251 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.026358 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:22.028919 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:22.028921 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:22.028921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:22.028929 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:22.036309 (+ 7380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:22.037745 (+ 1436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:22.037747 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:22.037747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:22.037816 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:22.038128 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:22.038129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:22.038129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:22.038237 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:22.038293 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:22.038496 (+ 203us) server_negotiation.cc:299] Negotiation successful
0929 19:03:22.038558 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 19:03:22.038852 15814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:22.022555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.124:46133 (local address 127.8.158.65:46625)
0929 19:03:22.022773 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:22.022781 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:22.023352 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:22.023617 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:22.023621 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:22.024084 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:22.024261 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:22.024266 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.025116 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.025116 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:22.025915 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:22.025916 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.026357 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.026358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:22.026429 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:22.026953 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:22.026958 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:22.028828 (+ 1870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:22.037486 (+ 8658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:22.037487 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:22.037488 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:22.037662 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:22.037980 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:22.037981 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:22.037981 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:22.038057 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:22.038310 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:22.038311 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:22.038389 (+ 78us) client_negotiation.cc:769] Sending connection context
0929 19:03:22.038478 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 19:03:22.038607 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":3,"thread_start_us":89,"threads_started":1}
W20260929 19:03:22.039336 15809 heartbeater.cc:644] Failed to heartbeat to 127.8.158.124:46133 (0 consecutive failures): Remote error: Failed to ping master at 127.8.158.124:46133: Service unavailable: service kudu.master.MasterService not registered on Master
W20260929 19:03:22.039717 15809 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 19:03:22.084647 15696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: false } } }
I20260929 19:03:22.084815 15696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260929 19:03:22.085264 15475 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f: attempting to promote NON_VOTER 9cee7e7f18ff4e888a7b900cc2049cbf to VOTER
I20260929 19:03:22.085448 15475 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: false } } peers { permanent_uuid: "9cee7e7f18ff4e888a7b900cc2049cbf" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46133 } attrs { promote: false } }
I20260929 19:03:22.086006 15535 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [term 1 FOLLOWER]: Refusing update from remote peer b3b7fb74082b41ab8c6bbdb28a50707f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 19:03:22.086119 15578 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260929 19:03:22.086258 15578 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260929 19:03:22.086230 15476 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 19:03:22.086301 15578 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf: stopping tablet replica
W20260929 19:03:22.086925 14900 consensus_peers.cc:487] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f -> Peer 9cee7e7f18ff4e888a7b900cc2049cbf (127.8.158.124:46133): Couldn't send request to peer 9cee7e7f18ff4e888a7b900cc2049cbf. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 19:03:22.086994 15476 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9cee7e7f18ff4e888a7b900cc2049cbf (127.8.158.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: false } } peers { permanent_uuid: "9cee7e7f18ff4e888a7b900cc2049cbf" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46133 } attrs { promote: false } } }
I20260929 19:03:22.087952 15476 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: false } } peers { permanent_uuid: "9cee7e7f18ff4e888a7b900cc2049cbf" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46133 } attrs { promote: false } } }
I20260929 19:03:22.088024 15476 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:22.088017 15474 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: false } } peers { permanent_uuid: "9cee7e7f18ff4e888a7b900cc2049cbf" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46133 } attrs { promote: false } } }
I20260929 19:03:22.088130 15474 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:22.088363 15535 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9cee7e7f18ff4e888a7b900cc2049cbf (127.8.158.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: false } } peers { permanent_uuid: "9cee7e7f18ff4e888a7b900cc2049cbf" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46133 } attrs { promote: false } } }
I20260929 19:03:22.101634 15578 master.cc:552] Master@127.8.158.124:46133 shutdown complete.
I20260929 19:03:22.101711 15578 master_runner.cc:305] Clearing existing system tablet
I20260929 19:03:22.101850 15578 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:22.101899 15578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:22.102753 15578 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 19:03:22.103268 15819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:22.103401 15578 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:22.103462 15578 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:03:22.103489 15578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-2/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-2/wal
uuid: "9cee7e7f18ff4e888a7b900cc2049cbf"
format_stamp: "Formatted at 2026-09-29 19:03:21 on dist-test-slave-nmsz"
server_key: "0e0cfb142280668e93d5a1c2073d9453"
server_key_iv: "096c4222247884c05fee363ad49f75e6"
server_key_version: "encryptionkey@0"
I20260929 19:03:22.104118 15578 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 19:03:22.125298 15816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:22.125504 15816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:22.125540 15816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:22.126719 15816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:22.126777 15816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:22.126830 15816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.66
I20260929 19:03:22.128321 15816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:40011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.66.keytab
--principal=oryx/127.8.158.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.66:42523
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.158.66
--webserver_port=42209
--tserver_master_addrs=127.8.158.126:37015,127.8.158.125:42397,127.8.158.124:46133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.66
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:22.128572 15816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:22.128784 15816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:22.133744 15827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:22.133765 15824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:22.133785 15825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:22.133986 15816 server_base.cc:1034] running on GCE node
I20260929 19:03:22.134200 15816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:22.134518 15816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:22.135702 15816 hybrid_clock.cc:648] HybridClock initialized: now 1790708602135689 us; error 27 us; skew 500 ppm
Sep 29 19:03:22 dist-test-slave-nmsz krb5kdc[14870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708602, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.158.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:22.138535 15816 init.cc:377] Logged in from keytab as oryx/127.8.158.66@KRBTEST.COM (short username oryx)
I20260929 19:03:22.139045 15816 webserver.cc:460] Webserver started at http://127.8.158.66:42209/ using document root <none> and password file <none>
I20260929 19:03:22.139302 15816 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:22.139389 15816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:22.141151 15816 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:03:22.141928 15834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:22.142093 15816 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:22.142174 15816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-1/wal
uuid: "a7a17ab10f8e4693b7d4321931c44da6"
format_stamp: "Formatted at 2026-09-29 19:03:20 on dist-test-slave-nmsz"
server_key: "5a0f7b8e9cd40c1dadfb19e93f0ed72e"
server_key_iv: "cb71e25b73c956748beae0a1c0c6fcca"
server_key_version: "encryptionkey@0"
I20260929 19:03:22.142596 15816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:22.147599 15564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: false } } peers { permanent_uuid: "9cee7e7f18ff4e888a7b900cc2049cbf" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46133 } attrs { promote: false } } }
I20260929 19:03:22.147703 15564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:03:22.148486 15578 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 19:03:22.148579 15578 log.cc:1196] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 19:03:22.148800 15578 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf: Deleting consensus metadata
I20260929 19:03:22.149044 15578 master_runner.cc:315] Copying system tablet from 127.8.158.126:37015
I20260929 19:03:22.149675 15578 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf: tablet copy: Beginning tablet copy session from remote peer at address 127.8.158.126:37015
I20260929 19:03:22.155561 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:22.150003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:37015 (local address 127.0.0.1:58594)
0929 19:03:22.150174 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:22.150189 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:22.150260 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:22.150421 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:22.150421 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:22.150597 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:22.150636 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:22.150638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.151290 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.151291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:22.151829 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:22.151831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.152199 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.152200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:22.152263 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:22.152718 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:22.152722 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:22.153079 (+ 357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:22.154549 (+ 1470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:22.154550 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:22.154550 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:22.154718 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:22.154927 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:22.154927 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:22.154928 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:22.154984 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:22.155234 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:22.155234 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:22.155277 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 19:03:22.155365 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 19:03:22.155466 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":88,"threads_started":1}
I20260929 19:03:22.155556 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:22.150067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58594 (local address 127.8.158.126:37015)
0929 19:03:22.150127 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:22.150129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:22.150255 (+ 126us) server_negotiation.cc:407] Connection header received
0929 19:03:22.150347 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:22.150348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:22.150359 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:22.150416 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:22.150726 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.151208 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.151912 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.152111 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.152196 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:22.153168 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:22.153169 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:22.153169 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:22.153177 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:22.154460 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:22.154801 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:22.154803 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:22.154803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:22.154862 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:22.155055 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:22.155056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:22.155056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:22.155166 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:22.155242 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:22.155390 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 19:03:22.155465 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 19:03:22.155756 14935 tablet_copy_service.cc:140] P b3b7fb74082b41ab8c6bbdb28a50707f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9cee7e7f18ff4e888a7b900cc2049cbf ({username='oryx', principal='oryx/127.8.158.124@KRBTEST.COM'} at 127.0.0.1:58594)
I20260929 19:03:22.155820 14935 tablet_copy_service.cc:161] P b3b7fb74082b41ab8c6bbdb28a50707f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9cee7e7f18ff4e888a7b900cc2049cbf at {username='oryx', principal='oryx/127.8.158.124@KRBTEST.COM'} at 127.0.0.1:58594: session id = 9cee7e7f18ff4e888a7b900cc2049cbf-00000000000000000000000000000000
I20260929 19:03:22.156222 14935 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 19:03:22.156658 15578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 19:03:22.157554 15578 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf: tablet copy: Starting download of 0 data blocks...
I20260929 19:03:22.157702 15578 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf: tablet copy: Starting download of 1 WAL segments...
I20260929 19:03:22.158421 15578 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 19:03:22.159168 14935 tablet_copy_service.cc:342] P b3b7fb74082b41ab8c6bbdb28a50707f: Request end of tablet copy session 9cee7e7f18ff4e888a7b900cc2049cbf-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.8.158.124@KRBTEST.COM'} at 127.0.0.1:58594
I20260929 19:03:22.159219 14935 tablet_copy_service.cc:433] P b3b7fb74082b41ab8c6bbdb28a50707f: ending tablet copy session 9cee7e7f18ff4e888a7b900cc2049cbf-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9cee7e7f18ff4e888a7b900cc2049cbf
I20260929 19:03:22.161504 15578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:22.162595 15846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:22.162650 15845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:22.162763 15848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:22.162854 15578 server_base.cc:1034] running on GCE node
I20260929 19:03:22.162936 15578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:22.163076 15578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:22.164232 15578 hybrid_clock.cc:648] HybridClock initialized: now 1790708602164222 us; error 23 us; skew 500 ppm
I20260929 19:03:22.164933 15816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
Sep 29 19:03:22 dist-test-slave-nmsz krb5kdc[14870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708602, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.158.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:22.165606 15578 init.cc:377] Logged in from keytab as oryx/127.8.158.124@KRBTEST.COM (short username oryx)
I20260929 19:03:22.165866 15578 webserver.cc:460] Webserver started at http://127.8.158.124:46875/ using document root <none> and password file <none>
I20260929 19:03:22.165985 15578 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:22.166023 15578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:22.167033 15578 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:22.167510 15816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:22.167650 15855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:22.167665 15816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:22.167848 15578 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:03:22.167904 15578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-2/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-2/wal
uuid: "9cee7e7f18ff4e888a7b900cc2049cbf"
format_stamp: "Formatted at 2026-09-29 19:03:21 on dist-test-slave-nmsz"
server_key: "0e0cfb142280668e93d5a1c2073d9453"
server_key_iv: "096c4222247884c05fee363ad49f75e6"
server_key_version: "encryptionkey@0"
I20260929 19:03:22.167924 15816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:22.167963 15578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:22.168243 15816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:22.168298 15816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:22.168332 15816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:22.168354 15816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:22.174782 15816 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.66:42523
I20260929 19:03:22.174970 15968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.66:42523 every 8 connection(s)
I20260929 19:03:22.175206 15816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
I20260929 19:03:22.176717 15975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:22.175897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.124:46133 (local address 127.8.158.66:57545)
0929 19:03:22.176243 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:22.176313 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.124:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":100,"threads_started":1}
W20260929 19:03:22.176966 15969 heartbeater.cc:644] Failed to heartbeat to 127.8.158.124:46133 (0 consecutive failures): Network error: Failed to ping master at 127.8.158.124:46133: Client connection negotiation failed: client connection to 127.8.158.124:46133: connect: Connection refused (error 111)
I20260929 19:03:22.177181 15578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:22.177851 15974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:22.177140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.124:46133 (local address 127.8.158.66:41585)
0929 19:03:22.177529 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:22.177560 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.124:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365}
I20260929 19:03:22.178563 15974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:22.178179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.124:46133 (local address 127.8.158.66:38983)
0929 19:03:22.178477 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:22.178505 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.124:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285}
I20260929 19:03:22.178920 15578 kserver.cc:163] Server-wide thread pool size limit: 3276
W20260929 19:03:22.179031 15969 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 19:03:22.181727 15578 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.124:46133
I20260929 19:03:22.182163 15578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/master-2/data/info.pb
Sep 29 19:03:22 dist-test-slave-nmsz krb5kdc[14870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708602, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.158.66@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM
I20260929 19:03:22.183211 16023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.124:46133 every 8 connection(s)
I20260929 19:03:22.183405 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 15816
Sep 29 19:03:22 dist-test-slave-nmsz krb5kdc[14870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708602, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.158.66@KRBTEST.COM for oryx/127.8.158.125@KRBTEST.COM
I20260929 19:03:22.183555 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 15245
I20260929 19:03:22.184515 16024 sys_catalog.cc:263] Verifying existing consensus state
I20260929 19:03:22.185063 16024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf: Bootstrap starting.
I20260929 19:03:22.187453 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:22.175926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:56985 (local address 127.8.158.126:37015)
0929 19:03:22.176025 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:22.176027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:22.176904 (+ 877us) server_negotiation.cc:407] Connection header received
0929 19:03:22.177551 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:22.177552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:22.177565 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:22.177625 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:22.178432 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.179133 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.180423 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.180741 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.180813 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:22.184283 (+ 3470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:22.184285 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:22.184285 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:22.184293 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:22.185935 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:22.186478 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:22.186479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:22.186480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:22.186541 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:22.186842 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:22.186843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:22.186843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:22.186946 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:22.186996 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:22.187327 (+ 331us) server_negotiation.cc:299] Negotiation successful
0929 19:03:22.187392 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 19:03:22.187625 15975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:22.175899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:37015 (local address 127.8.158.66:56985)
0929 19:03:22.176804 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:22.176820 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:22.177462 (+ 642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:22.177693 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:22.177697 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:22.178057 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:22.178358 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:22.178364 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.179286 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.179287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:22.180334 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:22.180336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.180820 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.180821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:22.180950 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:22.181457 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:22.181467 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:22.184191 (+ 2724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:22.186023 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:22.186036 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:22.186038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:22.186389 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:22.186615 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:22.186616 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:22.186616 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:22.186764 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:22.187010 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:22.187015 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:22.187181 (+ 166us) client_negotiation.cc:769] Sending connection context
0929 19:03:22.187286 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 19:03:22.187462 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":887,"thread_start_us":108,"threads_started":1}
I20260929 19:03:22.188090 15970 heartbeater.cc:347] Connected to a master server at 127.8.158.126:37015
I20260929 19:03:22.188156 15970 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:22.188292 15970 heartbeater.cc:511] Master 127.8.158.126:37015 requested a full tablet report, sending...
I20260929 19:03:22.188651 14915 ts_manager.cc:194] Re-registered known tserver with Master: a7a17ab10f8e4693b7d4321931c44da6 (127.8.158.66:42523)
I20260929 19:03:22.189136 14915 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.158.66@KRBTEST.COM'} at 127.8.158.66:56985
I20260929 19:03:22.190537 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.67:38403
--local_ip_for_outbound_sockets=127.8.158.67
--tserver_master_addrs=127.8.158.126:37015
--webserver_port=40249
--webserver_interface=127.8.158.67
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.67.keytab
--principal=oryx/127.8.158.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:40011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.158.126:37015,127.8.158.125:42397
--tserver_master_addrs=127.8.158.126:37015,127.8.158.125:42397,127.8.158.124:46133 with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:03:22.191617 16024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 19:03:22.191849 16024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf: Bootstrap complete.
I20260929 19:03:22.192036 16024 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: false } } peers { permanent_uuid: "9cee7e7f18ff4e888a7b900cc2049cbf" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46133 } attrs { promote: false } }
I20260929 19:03:22.192144 16024 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cee7e7f18ff4e888a7b900cc2049cbf, State: Initialized, Role: FOLLOWER
I20260929 19:03:22.192205 16024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: false } } peers { permanent_uuid: "9cee7e7f18ff4e888a7b900cc2049cbf" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46133 } attrs { promote: false } }
I20260929 19:03:22.192382 16024 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:03:22.193226 15576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:22.176068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:39021 (local address 127.8.158.125:42397)
0929 19:03:22.176109 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:22.176111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:22.177475 (+ 1364us) server_negotiation.cc:407] Connection header received
0929 19:03:22.178374 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:22.178375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:22.178388 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:22.178420 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:22.179165 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.179909 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.181074 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.181668 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.181737 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:22.190077 (+ 8340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:22.190078 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:22.190078 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:22.190085 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:22.191931 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:22.192376 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:22.192377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:22.192377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:22.192439 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:22.192650 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:22.192651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:22.192652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:22.192808 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:22.192874 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:22.193062 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 19:03:22.193133 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 19:03:22.193240 15973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:22.175953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.125:42397 (local address 127.8.158.66:39021)
0929 19:03:22.177394 (+ 1441us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:22.177404 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:22.177462 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:22.178698 (+ 1236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:22.178699 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:22.178874 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:22.178945 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:22.178948 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.179998 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.179999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:22.180982 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:22.180985 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.181795 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.181796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:22.181859 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:22.182290 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:22.182295 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:22.183900 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:22.192009 (+ 8109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:22.192010 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:22.192010 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:22.192252 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:22.192510 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:22.192511 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:22.192511 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:22.192584 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:22.192903 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:22.192904 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:22.192944 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 19:03:22.193017 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 19:03:22.193133 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1425,"mutex_wait_us":1,"thread_start_us":65,"threads_started":1}
I20260929 19:03:22.193681 15971 heartbeater.cc:347] Connected to a master server at 127.8.158.125:42397
I20260929 19:03:22.193729 15971 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:22.193686 16028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: false } } peers { permanent_uuid: "9cee7e7f18ff4e888a7b900cc2049cbf" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46133 } attrs { promote: false } } }
I20260929 19:03:22.193768 16028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:03:22.193799 15971 heartbeater.cc:511] Master 127.8.158.125:42397 requested a full tablet report, sending...
I20260929 19:03:22.194037 16039 catalog_manager.cc:1259] Loaded cluster ID: 7c8d5b9d44a245eba27c93910892b2b7
I20260929 19:03:22.194077 16039 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf: loading cluster ID for follower catalog manager: success
I20260929 19:03:22.194144 15525 ts_manager.cc:194] Registered new tserver with Master: a7a17ab10f8e4693b7d4321931c44da6 (127.8.158.66:42523)
I20260929 19:03:22.195377 16039 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf: acquiring CA information for follower catalog manager: success
I20260929 19:03:22.195596 16039 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 19:03:22.272061 16027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:22.272282 16027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:22.272318 16027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:22.273515 16027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:22.273576 16027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:22.273620 16027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.67
I20260929 19:03:22.275163 16027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:40011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.67.keytab
--principal=oryx/127.8.158.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.67:38403
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.158.67
--webserver_port=40249
--tserver_master_addrs=127.8.158.126:37015,127.8.158.125:42397,127.8.158.124:46133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.67
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:22.275405 16027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:22.275636 16027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:22.280601 16047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:22.280633 16027 server_base.cc:1034] running on GCE node
W20260929 19:03:22.280637 16045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:22.280614 16044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:22.281082 16027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:22.281309 16027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:22.282485 16027 hybrid_clock.cc:648] HybridClock initialized: now 1790708602282471 us; error 26 us; skew 500 ppm
Sep 29 19:03:22 dist-test-slave-nmsz krb5kdc[14870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708602, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.158.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:22.284981 16027 init.cc:377] Logged in from keytab as oryx/127.8.158.67@KRBTEST.COM (short username oryx)
I20260929 19:03:22.285334 16027 webserver.cc:460] Webserver started at http://127.8.158.67:40249/ using document root <none> and password file <none>
I20260929 19:03:22.285511 16027 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:22.285574 16027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:22.286919 16027 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:22.287451 16054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:22.287657 16027 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:22.287760 16027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-2/wal
uuid: "b57b14eb0e6a4cb5a5df9a09c73924e7"
format_stamp: "Formatted at 2026-09-29 19:03:20 on dist-test-slave-nmsz"
server_key: "e7063bc57b6bbeb11abe9a0dacf11347"
server_key_iv: "dd87072d70cf995320104be4b18cdc76"
server_key_version: "encryptionkey@0"
I20260929 19:03:22.288076 16027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:22.298848 16027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:22.301271 16027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:22.301427 16027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:22.301651 16027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:22.301940 16027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:22.301976 16027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:22.301996 16027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:22.302039 16027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:22.307648 16027 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.67:38403
I20260929 19:03:22.307717 16167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.67:38403 every 8 connection(s)
I20260929 19:03:22.308063 16027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
Sep 29 19:03:22 dist-test-slave-nmsz krb5kdc[14870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708602, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.158.67@KRBTEST.COM for oryx/127.8.158.124@KRBTEST.COM
Sep 29 19:03:22 dist-test-slave-nmsz krb5kdc[14870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708602, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.158.67@KRBTEST.COM for oryx/127.8.158.125@KRBTEST.COM
Sep 29 19:03:22 dist-test-slave-nmsz krb5kdc[14870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708602, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.158.67@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM
I20260929 19:03:22.316635 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 16027
I20260929 19:03:22.319221 15576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:22.308758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:46307 (local address 127.8.158.125:42397)
0929 19:03:22.308860 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:22.308863 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:22.309247 (+ 384us) server_negotiation.cc:407] Connection header received
0929 19:03:22.309726 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:22.309728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:22.309740 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:22.309818 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:22.310705 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.311526 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.312497 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.312786 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.312849 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:22.315823 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:22.315824 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:22.315825 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:22.315832 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:22.317564 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:22.318143 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:22.318145 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:22.318145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:22.318201 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:22.318511 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:22.318512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:22.318512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:22.318628 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:22.318678 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:22.319050 (+ 372us) server_negotiation.cc:299] Negotiation successful
0929 19:03:22.319129 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 19:03:22.319449 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:22.308810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:34713 (local address 127.8.158.126:37015)
0929 19:03:22.309028 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:22.309031 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:22.309918 (+ 887us) server_negotiation.cc:407] Connection header received
0929 19:03:22.309947 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:22.309948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:22.309960 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:22.310032 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:22.310734 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.311254 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.312498 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.312809 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.312850 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:22.316659 (+ 3809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:22.316660 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:22.316661 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:22.316668 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:22.318366 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:22.318726 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:22.318727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:22.318727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:22.318778 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:22.318974 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:22.318975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:22.318976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:22.319093 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:22.319154 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:22.319284 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 19:03:22.319351 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177}
I20260929 19:03:22.319458 16172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:22.308730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.125:42397 (local address 127.8.158.67:46307)
0929 19:03:22.309133 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:22.309163 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:22.309627 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:22.309922 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:22.309926 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:22.310284 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:22.310611 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:22.310615 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.311617 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.311618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:22.312398 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:22.312400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.312853 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.312854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:22.313012 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:22.313650 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:22.313660 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:22.315758 (+ 2098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:22.317654 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:22.317661 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:22.317663 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:22.318055 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:22.318269 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:22.318270 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:22.318271 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:22.318450 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:22.318700 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:22.318706 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:22.318859 (+ 153us) client_negotiation.cc:769] Sending connection context
0929 19:03:22.318962 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 19:03:22.319180 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":98,"thread_start_us":83,"threads_started":1}
I20260929 19:03:22.319463 16173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:22.308731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:37015 (local address 127.8.158.67:34713)
0929 19:03:22.309816 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:22.309826 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:22.309876 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:22.310062 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:22.310063 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:22.310375 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:22.310613 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:22.310619 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.311348 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.311349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:22.312401 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:22.312403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.312925 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.312926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:22.313011 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:22.314024 (+ 1013us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:22.314029 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:22.316586 (+ 2557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:22.318425 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:22.318426 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:22.318427 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:22.318643 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:22.318850 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:22.318850 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:22.318851 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:22.318909 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:22.319145 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:22.319146 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:22.319182 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 19:03:22.319249 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 19:03:22.319343 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1059,"mutex_wait_us":6,"thread_start_us":106,"threads_started":1}
I20260929 19:03:22.319911 16169 heartbeater.cc:347] Connected to a master server at 127.8.158.126:37015
I20260929 19:03:22.320005 16175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:22.308909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:43859 (local address 127.8.158.124:46133)
0929 19:03:22.309074 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:22.309076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:22.310373 (+ 1297us) server_negotiation.cc:407] Connection header received
0929 19:03:22.310401 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:22.310403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:22.310414 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:22.310479 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:22.310807 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.311770 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.312768 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.313069 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.313131 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:22.315502 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:22.315510 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:22.315510 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:22.315518 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:22.317606 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:22.318119 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:22.318120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:22.318120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:22.318176 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:22.318538 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:22.318539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:22.318539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:22.318672 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:22.318732 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:22.319657 (+ 925us) server_negotiation.cc:299] Negotiation successful
0929 19:03:22.319895 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":60,"threads_started":1}
I20260929 19:03:22.319998 16170 heartbeater.cc:347] Connected to a master server at 127.8.158.125:42397
I20260929 19:03:22.320009 16169 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:22.320055 16170 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:22.320205 16169 heartbeater.cc:511] Master 127.8.158.126:37015 requested a full tablet report, sending...
I20260929 19:03:22.320271 16174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:22.308731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.124:46133 (local address 127.8.158.67:43859)
0929 19:03:22.310242 (+ 1511us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:22.310252 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:22.310327 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:22.310509 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:22.310510 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:22.310668 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:22.310728 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:22.310729 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.311854 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.311855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:22.312699 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:22.312701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.313130 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.313131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:22.313223 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:22.313813 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:22.313820 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:22.315393 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:22.317671 (+ 2278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:22.317672 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:22.317673 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:22.318033 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:22.318250 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:22.318251 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:22.318251 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:22.318445 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:22.318733 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:22.318733 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:22.318859 (+ 126us) client_negotiation.cc:769] Sending connection context
0929 19:03:22.318959 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 19:03:22.320144 (+ 1185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1196,"mutex_wait_us":61,"thread_start_us":56,"threads_started":1}
I20260929 19:03:22.320211 16170 heartbeater.cc:511] Master 127.8.158.125:42397 requested a full tablet report, sending...
I20260929 19:03:22.320556 14915 ts_manager.cc:194] Re-registered known tserver with Master: b57b14eb0e6a4cb5a5df9a09c73924e7 (127.8.158.67:38403)
I20260929 19:03:22.320847 15525 ts_manager.cc:194] Registered new tserver with Master: b57b14eb0e6a4cb5a5df9a09c73924e7 (127.8.158.67:38403)
I20260929 19:03:22.321064 14915 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.158.67@KRBTEST.COM'} at 127.8.158.67:34713
I20260929 19:03:22.321180 16168 heartbeater.cc:347] Connected to a master server at 127.8.158.124:46133
I20260929 19:03:22.321226 16168 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:22.321309 16168 heartbeater.cc:511] Master 127.8.158.124:46133 requested a full tablet report, sending...
I20260929 19:03:22.321779 15988 ts_manager.cc:194] Registered new tserver with Master: b57b14eb0e6a4cb5a5df9a09c73924e7 (127.8.158.67:38403)
I20260929 19:03:22.329464 16175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:22.321364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58208 (local address 127.8.158.124:46133)
0929 19:03:22.321433 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:22.321435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:22.322214 (+ 779us) server_negotiation.cc:407] Connection header received
0929 19:03:22.322298 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:22.322299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:22.322308 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:22.322390 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:22.323021 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.323674 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.324387 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.324668 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.324725 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:22.326231 (+ 1506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:22.326232 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:22.326233 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:22.326240 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:22.328266 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:22.328723 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:22.328723 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:22.328724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:22.328773 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:22.328963 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:22.328964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:22.328964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:22.329092 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:22.329182 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:22.329281 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 19:03:22.329348 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 19:03:22.329806 15576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:22.321304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56944 (local address 127.8.158.125:42397)
0929 19:03:22.321404 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:22.321406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:22.321680 (+ 274us) server_negotiation.cc:407] Connection header received
0929 19:03:22.321766 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:22.321767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:22.321777 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:22.321829 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:22.322744 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.323475 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.324446 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.324738 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.324779 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:22.326918 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:22.326919 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:22.326920 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:22.326926 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:22.328605 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:22.328990 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:22.328991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:22.328991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:22.329035 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:22.329211 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:22.329213 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:22.329213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:22.329335 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:22.329411 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:22.329612 (+ 201us) server_negotiation.cc:299] Negotiation successful
0929 19:03:22.329678 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 19:03:22.329896 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:22.321483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58606 (local address 127.8.158.126:37015)
0929 19:03:22.321639 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:22.321640 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:22.322048 (+ 408us) server_negotiation.cc:407] Connection header received
0929 19:03:22.322111 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:22.322112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:22.322122 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:22.322174 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:22.322903 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.323359 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.324493 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.324772 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.324830 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:22.326697 (+ 1867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:22.326699 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:22.326699 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:22.326706 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:22.328294 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:22.328797 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:22.328798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:22.328798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:22.328857 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:22.329045 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:22.329047 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:22.329047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:22.329184 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:22.329273 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:22.329458 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 19:03:22.329771 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260929 19:03:22.331429 14915 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58606:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:03:22.332947 14915 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:03:22.340026 16187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:22.336142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.66:42523 (local address 127.0.0.1:46542)
0929 19:03:22.336271 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:22.336282 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:22.336331 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:22.336837 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:22.336837 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:22.336839 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:22.336892 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:22.336893 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.338153 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.338154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:22.339040 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:22.339042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.339781 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.339782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:22.339839 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:22.339844 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 19:03:22.339894 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 19:03:22.339930 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":26,"thread_start_us":75,"threads_started":1}
I20260929 19:03:22.340121 15502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:22.336114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.67:38403 (local address 127.0.0.1:48292)
0929 19:03:22.336159 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:22.336169 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:22.336206 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:22.336624 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:22.336625 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:22.336627 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:22.336675 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:22.336677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.337632 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.337633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:22.338514 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:22.338516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.339882 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.339883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:22.339936 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:22.339938 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 19:03:22.339990 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 19:03:22.340022 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 19:03:22.340332 16188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:22.336235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48292 (local address 127.8.158.67:38403)
0929 19:03:22.336424 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:22.336428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:22.336438 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:03:22.336476 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:22.336478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:22.336493 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:22.336747 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:22.336797 (+ 50us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.337548 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.339119 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.339789 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.339877 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:22.339945 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:22.340009 (+ 64us) server_negotiation.cc:299] Negotiation successful
0929 19:03:22.340057 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":96,"threads_started":1}
I20260929 19:03:22.340363 16190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:22.336417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46542 (local address 127.8.158.66:42523)
0929 19:03:22.336666 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:22.336670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:22.336679 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:03:22.336715 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:22.336717 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:22.336733 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:22.336837 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:22.336974 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.338064 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.339117 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.339692 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.339781 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:22.339837 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:22.340199 (+ 362us) server_negotiation.cc:299] Negotiation successful
0929 19:03:22.340250 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":94,"threads_started":1}
I20260929 19:03:22.340420 16189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:22.336250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:43095 (local address 127.0.0.1:38064)
0929 19:03:22.336408 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:22.336417 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:22.336689 (+ 272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:22.336939 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:22.336940 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:22.336942 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:22.336978 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:22.336979 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.338164 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.338165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:22.339176 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:22.339179 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.340162 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.340163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:22.340232 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:22.340236 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:03:22.340287 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 19:03:22.340334 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":25,"thread_start_us":112,"threads_started":1}
I20260929 19:03:22.340687 16191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:22.336494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38064 (local address 127.8.158.65:43095)
0929 19:03:22.336627 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:22.336631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:22.336642 (+ 11us) server_negotiation.cc:407] Connection header received
0929 19:03:22.336844 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:22.336846 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:22.336862 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:22.336942 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:22.337049 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.338062 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.339265 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.340000 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.340061 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:22.340128 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:22.340567 (+ 439us) server_negotiation.cc:299] Negotiation successful
0929 19:03:22.340613 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":72,"threads_started":1}
I20260929 19:03:22.340941 16102 tablet_service.cc:1467] Processing CreateTablet for tablet 2909c1748d8840e2afbb0f32593c93d7 (DEFAULT_TABLE table=test-table [id=fdbfefa85bc740599800ddd0535c6b3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:22.340947 15903 tablet_service.cc:1467] Processing CreateTablet for tablet 2909c1748d8840e2afbb0f32593c93d7 (DEFAULT_TABLE table=test-table [id=fdbfefa85bc740599800ddd0535c6b3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:22.341197 16102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2909c1748d8840e2afbb0f32593c93d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:22.341303 15903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2909c1748d8840e2afbb0f32593c93d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:22.343995 16192 tablet_bootstrap.cc:492] T 2909c1748d8840e2afbb0f32593c93d7 P b57b14eb0e6a4cb5a5df9a09c73924e7: Bootstrap starting.
I20260929 19:03:22.344276 16193 tablet_bootstrap.cc:492] T 2909c1748d8840e2afbb0f32593c93d7 P a7a17ab10f8e4693b7d4321931c44da6: Bootstrap starting.
I20260929 19:03:22.344646 16192 tablet_bootstrap.cc:654] T 2909c1748d8840e2afbb0f32593c93d7 P b57b14eb0e6a4cb5a5df9a09c73924e7: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:22.344621 15743 tablet_service.cc:1467] Processing CreateTablet for tablet 2909c1748d8840e2afbb0f32593c93d7 (DEFAULT_TABLE table=test-table [id=fdbfefa85bc740599800ddd0535c6b3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:22.344913 16193 tablet_bootstrap.cc:654] T 2909c1748d8840e2afbb0f32593c93d7 P a7a17ab10f8e4693b7d4321931c44da6: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:22.344944 16192 log.cc:824] T 2909c1748d8840e2afbb0f32593c93d7 P b57b14eb0e6a4cb5a5df9a09c73924e7: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:22.345007 15743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2909c1748d8840e2afbb0f32593c93d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:22.345211 16193 log.cc:824] T 2909c1748d8840e2afbb0f32593c93d7 P a7a17ab10f8e4693b7d4321931c44da6: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:22.345778 16192 tablet_bootstrap.cc:492] T 2909c1748d8840e2afbb0f32593c93d7 P b57b14eb0e6a4cb5a5df9a09c73924e7: No bootstrap required, opened a new log
I20260929 19:03:22.345850 16192 ts_tablet_manager.cc:1397] T 2909c1748d8840e2afbb0f32593c93d7 P b57b14eb0e6a4cb5a5df9a09c73924e7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:03:22.345953 16193 tablet_bootstrap.cc:492] T 2909c1748d8840e2afbb0f32593c93d7 P a7a17ab10f8e4693b7d4321931c44da6: No bootstrap required, opened a new log
I20260929 19:03:22.346019 16193 ts_tablet_manager.cc:1397] T 2909c1748d8840e2afbb0f32593c93d7 P a7a17ab10f8e4693b7d4321931c44da6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:03:22.347729 16192 raft_consensus.cc:348] T 2909c1748d8840e2afbb0f32593c93d7 P b57b14eb0e6a4cb5a5df9a09c73924e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57b14eb0e6a4cb5a5df9a09c73924e7" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38403 } } peers { permanent_uuid: "a7a17ab10f8e4693b7d4321931c44da6" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 42523 } } peers { permanent_uuid: "cc3c76ee46724a5cb427c2f4ac12ab34" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 43095 } }
I20260929 19:03:22.347872 16192 raft_consensus.cc:374] T 2909c1748d8840e2afbb0f32593c93d7 P b57b14eb0e6a4cb5a5df9a09c73924e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:22.347906 16192 raft_consensus.cc:729] T 2909c1748d8840e2afbb0f32593c93d7 P b57b14eb0e6a4cb5a5df9a09c73924e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b57b14eb0e6a4cb5a5df9a09c73924e7, State: Initialized, Role: FOLLOWER
I20260929 19:03:22.348034 16192 consensus_queue.cc:260] T 2909c1748d8840e2afbb0f32593c93d7 P b57b14eb0e6a4cb5a5df9a09c73924e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57b14eb0e6a4cb5a5df9a09c73924e7" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38403 } } peers { permanent_uuid: "a7a17ab10f8e4693b7d4321931c44da6" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 42523 } } peers { permanent_uuid: "cc3c76ee46724a5cb427c2f4ac12ab34" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 43095 } }
I20260929 19:03:22.348009 16193 raft_consensus.cc:348] T 2909c1748d8840e2afbb0f32593c93d7 P a7a17ab10f8e4693b7d4321931c44da6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57b14eb0e6a4cb5a5df9a09c73924e7" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38403 } } peers { permanent_uuid: "a7a17ab10f8e4693b7d4321931c44da6" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 42523 } } peers { permanent_uuid: "cc3c76ee46724a5cb427c2f4ac12ab34" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 43095 } }
I20260929 19:03:22.348147 16193 raft_consensus.cc:374] T 2909c1748d8840e2afbb0f32593c93d7 P a7a17ab10f8e4693b7d4321931c44da6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:22.348188 16193 raft_consensus.cc:729] T 2909c1748d8840e2afbb0f32593c93d7 P a7a17ab10f8e4693b7d4321931c44da6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7a17ab10f8e4693b7d4321931c44da6, State: Initialized, Role: FOLLOWER
I20260929 19:03:22.348300 16192 ts_tablet_manager.cc:1428] T 2909c1748d8840e2afbb0f32593c93d7 P b57b14eb0e6a4cb5a5df9a09c73924e7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 19:03:22.348294 16193 consensus_queue.cc:260] T 2909c1748d8840e2afbb0f32593c93d7 P a7a17ab10f8e4693b7d4321931c44da6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57b14eb0e6a4cb5a5df9a09c73924e7" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38403 } } peers { permanent_uuid: "a7a17ab10f8e4693b7d4321931c44da6" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 42523 } } peers { permanent_uuid: "cc3c76ee46724a5cb427c2f4ac12ab34" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 43095 } }
I20260929 19:03:22.348497 16197 tablet_bootstrap.cc:492] T 2909c1748d8840e2afbb0f32593c93d7 P cc3c76ee46724a5cb427c2f4ac12ab34: Bootstrap starting.
I20260929 19:03:22.348531 16193 ts_tablet_manager.cc:1428] T 2909c1748d8840e2afbb0f32593c93d7 P a7a17ab10f8e4693b7d4321931c44da6: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260929 19:03:22.348738 15970 heartbeater.cc:503] Master 127.8.158.126:37015 was elected leader, sending a full tablet report...
I20260929 19:03:22.348907 16169 heartbeater.cc:503] Master 127.8.158.126:37015 was elected leader, sending a full tablet report...
I20260929 19:03:22.350916 16197 tablet_bootstrap.cc:654] T 2909c1748d8840e2afbb0f32593c93d7 P cc3c76ee46724a5cb427c2f4ac12ab34: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:22.351214 16197 log.cc:824] T 2909c1748d8840e2afbb0f32593c93d7 P cc3c76ee46724a5cb427c2f4ac12ab34: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:22.352069 16197 tablet_bootstrap.cc:492] T 2909c1748d8840e2afbb0f32593c93d7 P cc3c76ee46724a5cb427c2f4ac12ab34: No bootstrap required, opened a new log
I20260929 19:03:22.352123 16197 ts_tablet_manager.cc:1397] T 2909c1748d8840e2afbb0f32593c93d7 P cc3c76ee46724a5cb427c2f4ac12ab34: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260929 19:03:22.352512 16175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:22.349283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:59055 (local address 127.8.158.124:46133)
0929 19:03:22.349338 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:22.349340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:22.349347 (+ 7us) server_negotiation.cc:407] Connection header received
0929 19:03:22.349368 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:22.349369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:22.349385 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:22.349450 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:22.349583 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.350280 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.351665 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.352207 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.352274 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:22.352330 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:22.352435 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 19:03:22.352461 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 19:03:22.352551 15973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:22.348973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.124:46133 (local address 127.8.158.66:59055)
0929 19:03:22.349094 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:22.349105 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:22.349140 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:22.349445 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:22.349446 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:22.349448 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:22.349507 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:22.349509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.350397 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.350398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:22.351576 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:22.351579 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.352273 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.352274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:22.352347 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:22.352352 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 19:03:22.352423 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 19:03:22.352482 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104}
I20260929 19:03:22.352911 15969 heartbeater.cc:347] Connected to a master server at 127.8.158.124:46133
I20260929 19:03:22.352962 15969 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:22.353029 15969 heartbeater.cc:511] Master 127.8.158.124:46133 requested a full tablet report, sending...
I20260929 19:03:22.353334 15988 ts_manager.cc:194] Registered new tserver with Master: a7a17ab10f8e4693b7d4321931c44da6 (127.8.158.66:42523)
I20260929 19:03:22.354552 16197 raft_consensus.cc:348] T 2909c1748d8840e2afbb0f32593c93d7 P cc3c76ee46724a5cb427c2f4ac12ab34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57b14eb0e6a4cb5a5df9a09c73924e7" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38403 } } peers { permanent_uuid: "a7a17ab10f8e4693b7d4321931c44da6" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 42523 } } peers { permanent_uuid: "cc3c76ee46724a5cb427c2f4ac12ab34" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 43095 } }
I20260929 19:03:22.354691 16197 raft_consensus.cc:374] T 2909c1748d8840e2afbb0f32593c93d7 P cc3c76ee46724a5cb427c2f4ac12ab34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:22.354733 16197 raft_consensus.cc:729] T 2909c1748d8840e2afbb0f32593c93d7 P cc3c76ee46724a5cb427c2f4ac12ab34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc3c76ee46724a5cb427c2f4ac12ab34, State: Initialized, Role: FOLLOWER
I20260929 19:03:22.354867 16197 consensus_queue.cc:260] T 2909c1748d8840e2afbb0f32593c93d7 P cc3c76ee46724a5cb427c2f4ac12ab34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57b14eb0e6a4cb5a5df9a09c73924e7" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38403 } } peers { permanent_uuid: "a7a17ab10f8e4693b7d4321931c44da6" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 42523 } } peers { permanent_uuid: "cc3c76ee46724a5cb427c2f4ac12ab34" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 43095 } }
I20260929 19:03:22.355109 16197 ts_tablet_manager.cc:1428] T 2909c1748d8840e2afbb0f32593c93d7 P cc3c76ee46724a5cb427c2f4ac12ab34: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 19:03:22.355201 15810 heartbeater.cc:503] Master 127.8.158.126:37015 was elected leader, sending a full tablet report...
I20260929 19:03:22.358155 16175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:22.355558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:45929 (local address 127.8.158.124:46133)
0929 19:03:22.355711 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:22.355714 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:22.355722 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:03:22.355744 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:22.355745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:22.355755 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:22.355805 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:22.356031 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.356602 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.357598 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.357908 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.357964 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:22.357971 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:22.358097 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 19:03:22.358115 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260929 19:03:22.358166 15814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:22.355528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.124:46133 (local address 127.8.158.65:45929)
0929 19:03:22.355568 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:22.355579 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:22.355620 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:22.355824 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:22.355825 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:22.355827 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:22.355959 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:22.355961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.356667 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.356668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:22.357526 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:22.357529 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.357979 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.357980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:22.358040 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:22.358043 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:03:22.358093 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 19:03:22.358119 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260929 19:03:22.358618 15809 heartbeater.cc:347] Connected to a master server at 127.8.158.124:46133
I20260929 19:03:22.358671 15809 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:22.358708 15809 heartbeater.cc:511] Master 127.8.158.124:46133 requested a full tablet report, sending...
I20260929 19:03:22.358963 15988 ts_manager.cc:194] Registered new tserver with Master: cc3c76ee46724a5cb427c2f4ac12ab34 (127.8.158.65:43095)
W20260929 19:03:22.425983 15972 tablet.cc:2369] T 2909c1748d8840e2afbb0f32593c93d7 P a7a17ab10f8e4693b7d4321931c44da6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:03:22.428905 16196 raft_consensus.cc:482] T 2909c1748d8840e2afbb0f32593c93d7 P b57b14eb0e6a4cb5a5df9a09c73924e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:03:22.429045 16196 raft_consensus.cc:504] T 2909c1748d8840e2afbb0f32593c93d7 P b57b14eb0e6a4cb5a5df9a09c73924e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57b14eb0e6a4cb5a5df9a09c73924e7" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38403 } } peers { permanent_uuid: "a7a17ab10f8e4693b7d4321931c44da6" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 42523 } } peers { permanent_uuid: "cc3c76ee46724a5cb427c2f4ac12ab34" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 43095 } }
I20260929 19:03:22.429570 16196 leader_election.cc:288] T 2909c1748d8840e2afbb0f32593c93d7 P b57b14eb0e6a4cb5a5df9a09c73924e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7a17ab10f8e4693b7d4321931c44da6 (127.8.158.66:42523), cc3c76ee46724a5cb427c2f4ac12ab34 (127.8.158.65:43095)
I20260929 19:03:22.432600 16174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:22.429867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:43095 (local address 127.8.158.67:53741)
0929 19:03:22.429932 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:22.429945 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:22.430000 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:22.430236 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:22.430237 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:22.430239 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:22.430359 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:22.430361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.431050 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.431051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:22.431761 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:22.431763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.432331 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.432332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:22.432414 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:22.432417 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:03:22.432462 (+ 45us) client_negotiation.cc:240] Negotiation successful
0929 19:03:22.432489 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 19:03:22.432606 16191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:22.430024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:53741 (local address 127.8.158.65:43095)
0929 19:03:22.430100 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:22.430102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:22.430111 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:03:22.430138 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:22.430139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:22.430150 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:22.430232 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:22.430460 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.430960 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.431852 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.432247 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.432326 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:22.432337 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:22.432491 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 19:03:22.432535 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 19:03:22.432653 16190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:22.429965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:51449 (local address 127.8.158.66:42523)
0929 19:03:22.430008 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:22.430010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:22.430116 (+ 106us) server_negotiation.cc:407] Connection header received
0929 19:03:22.430155 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:22.430156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:22.430164 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:22.430223 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:22.430436 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.430901 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.431907 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.432253 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.432340 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:22.432348 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:22.432540 (+ 192us) server_negotiation.cc:299] Negotiation successful
0929 19:03:22.432580 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 19:03:22.432695 16173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:22.429867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.66:42523 (local address 127.8.158.67:51449)
0929 19:03:22.430031 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:22.430041 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:22.430102 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:22.430244 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:22.430244 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:22.430246 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:22.430362 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:22.430363 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.430983 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.430984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:22.431789 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:22.431791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.432355 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.432355 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:22.432439 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:22.432442 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:03:22.432519 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 19:03:22.432553 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":195}
I20260929 19:03:22.433024 15763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2909c1748d8840e2afbb0f32593c93d7" candidate_uuid: "b57b14eb0e6a4cb5a5df9a09c73924e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc3c76ee46724a5cb427c2f4ac12ab34" is_pre_election: true
I20260929 19:03:22.433051 15923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2909c1748d8840e2afbb0f32593c93d7" candidate_uuid: "b57b14eb0e6a4cb5a5df9a09c73924e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7a17ab10f8e4693b7d4321931c44da6" is_pre_election: true
I20260929 19:03:22.433171 15763 raft_consensus.cc:2454] T 2909c1748d8840e2afbb0f32593c93d7 P cc3c76ee46724a5cb427c2f4ac12ab34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b57b14eb0e6a4cb5a5df9a09c73924e7 in term 0.
I20260929 19:03:22.433193 15923 raft_consensus.cc:2454] T 2909c1748d8840e2afbb0f32593c93d7 P a7a17ab10f8e4693b7d4321931c44da6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b57b14eb0e6a4cb5a5df9a09c73924e7 in term 0.
I20260929 19:03:22.433436 16056 leader_election.cc:302] T 2909c1748d8840e2afbb0f32593c93d7 P b57b14eb0e6a4cb5a5df9a09c73924e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7a17ab10f8e4693b7d4321931c44da6, b57b14eb0e6a4cb5a5df9a09c73924e7; no voters:
I20260929 19:03:22.433616 16196 raft_consensus.cc:2781] T 2909c1748d8840e2afbb0f32593c93d7 P b57b14eb0e6a4cb5a5df9a09c73924e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:03:22.433697 16196 raft_consensus.cc:482] T 2909c1748d8840e2afbb0f32593c93d7 P b57b14eb0e6a4cb5a5df9a09c73924e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:03:22.433730 16196 raft_consensus.cc:3037] T 2909c1748d8840e2afbb0f32593c93d7 P b57b14eb0e6a4cb5a5df9a09c73924e7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:22.434437 16196 raft_consensus.cc:504] T 2909c1748d8840e2afbb0f32593c93d7 P b57b14eb0e6a4cb5a5df9a09c73924e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57b14eb0e6a4cb5a5df9a09c73924e7" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38403 } } peers { permanent_uuid: "a7a17ab10f8e4693b7d4321931c44da6" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 42523 } } peers { permanent_uuid: "cc3c76ee46724a5cb427c2f4ac12ab34" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 43095 } }
I20260929 19:03:22.434576 16196 leader_election.cc:288] T 2909c1748d8840e2afbb0f32593c93d7 P b57b14eb0e6a4cb5a5df9a09c73924e7 [CANDIDATE]: Term 1 election: Requested vote from peers a7a17ab10f8e4693b7d4321931c44da6 (127.8.158.66:42523), cc3c76ee46724a5cb427c2f4ac12ab34 (127.8.158.65:43095)
I20260929 19:03:22.434715 15923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2909c1748d8840e2afbb0f32593c93d7" candidate_uuid: "b57b14eb0e6a4cb5a5df9a09c73924e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7a17ab10f8e4693b7d4321931c44da6"
I20260929 19:03:22.434715 15763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2909c1748d8840e2afbb0f32593c93d7" candidate_uuid: "b57b14eb0e6a4cb5a5df9a09c73924e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc3c76ee46724a5cb427c2f4ac12ab34"
I20260929 19:03:22.434788 15923 raft_consensus.cc:3037] T 2909c1748d8840e2afbb0f32593c93d7 P a7a17ab10f8e4693b7d4321931c44da6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:22.434789 15763 raft_consensus.cc:3037] T 2909c1748d8840e2afbb0f32593c93d7 P cc3c76ee46724a5cb427c2f4ac12ab34 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:22.435573 15763 raft_consensus.cc:2454] T 2909c1748d8840e2afbb0f32593c93d7 P cc3c76ee46724a5cb427c2f4ac12ab34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b57b14eb0e6a4cb5a5df9a09c73924e7 in term 1.
I20260929 19:03:22.435573 15923 raft_consensus.cc:2454] T 2909c1748d8840e2afbb0f32593c93d7 P a7a17ab10f8e4693b7d4321931c44da6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b57b14eb0e6a4cb5a5df9a09c73924e7 in term 1.
I20260929 19:03:22.435765 16057 leader_election.cc:302] T 2909c1748d8840e2afbb0f32593c93d7 P b57b14eb0e6a4cb5a5df9a09c73924e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b57b14eb0e6a4cb5a5df9a09c73924e7, cc3c76ee46724a5cb427c2f4ac12ab34; no voters:
I20260929 19:03:22.435928 16196 raft_consensus.cc:2781] T 2909c1748d8840e2afbb0f32593c93d7 P b57b14eb0e6a4cb5a5df9a09c73924e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:03:22.436105 16196 raft_consensus.cc:686] T 2909c1748d8840e2afbb0f32593c93d7 P b57b14eb0e6a4cb5a5df9a09c73924e7 [term 1 LEADER]: Becoming Leader. State: Replica: b57b14eb0e6a4cb5a5df9a09c73924e7, State: Running, Role: LEADER
I20260929 19:03:22.436229 16196 consensus_queue.cc:237] T 2909c1748d8840e2afbb0f32593c93d7 P b57b14eb0e6a4cb5a5df9a09c73924e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57b14eb0e6a4cb5a5df9a09c73924e7" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38403 } } peers { permanent_uuid: "a7a17ab10f8e4693b7d4321931c44da6" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 42523 } } peers { permanent_uuid: "cc3c76ee46724a5cb427c2f4ac12ab34" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 43095 } }
I20260929 19:03:22.437027 14914 catalog_manager.cc:5495] T 2909c1748d8840e2afbb0f32593c93d7 P b57b14eb0e6a4cb5a5df9a09c73924e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to b57b14eb0e6a4cb5a5df9a09c73924e7 (127.8.158.67). New cstate: current_term: 1 leader_uuid: "b57b14eb0e6a4cb5a5df9a09c73924e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57b14eb0e6a4cb5a5df9a09c73924e7" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7a17ab10f8e4693b7d4321931c44da6" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 42523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc3c76ee46724a5cb427c2f4ac12ab34" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 43095 } health_report { overall_health: UNKNOWN } } }
I20260929 19:03:22.454133 16188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:22.451880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48306 (local address 127.8.158.67:38403)
0929 19:03:22.451937 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:22.451939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:22.451947 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:03:22.451979 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:22.451980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:22.452000 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:22.452065 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:03:22.452178 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.452627 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.453419 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.453657 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.453724 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:22.453861 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:03:22.453960 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:03:22.454026 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:22.454061 (+ 35us) server_negotiation.cc:299] Negotiation successful
0929 19:03:22.454078 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 19:03:22.455827 15763 raft_consensus.cc:1261] T 2909c1748d8840e2afbb0f32593c93d7 P cc3c76ee46724a5cb427c2f4ac12ab34 [term 1 FOLLOWER]: Refusing update from remote peer b57b14eb0e6a4cb5a5df9a09c73924e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:03:22.455855 15923 raft_consensus.cc:1261] T 2909c1748d8840e2afbb0f32593c93d7 P a7a17ab10f8e4693b7d4321931c44da6 [term 1 FOLLOWER]: Refusing update from remote peer b57b14eb0e6a4cb5a5df9a09c73924e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:03:22.456185 16201 consensus_queue.cc:1035] T 2909c1748d8840e2afbb0f32593c93d7 P b57b14eb0e6a4cb5a5df9a09c73924e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3c76ee46724a5cb427c2f4ac12ab34" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 43095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:03:22.456326 16196 consensus_queue.cc:1035] T 2909c1748d8840e2afbb0f32593c93d7 P b57b14eb0e6a4cb5a5df9a09c73924e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7a17ab10f8e4693b7d4321931c44da6" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 42523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:03:22.459725 16205 mvcc.cc:205] Tried to move back new op lower bound from 7334742435656769536 to 7334742435579551744. Current Snapshot: MvccSnapshot[applied={T|T < 7334742435656769536}]
I20260929 19:03:22.459872 16204 mvcc.cc:205] Tried to move back new op lower bound from 7334742435656769536 to 7334742435579551744. Current Snapshot: MvccSnapshot[applied={T|T < 7334742435656769536}]
I20260929 19:03:22.460069 16206 mvcc.cc:205] Tried to move back new op lower bound from 7334742435656769536 to 7334742435579551744. Current Snapshot: MvccSnapshot[applied={T|T < 7334742435656769536}]
I20260929 19:03:22.462450 14915 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:03:22.462523 14915 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:03:22.463773 14915 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b3b7fb74082b41ab8c6bbdb28a50707f: Sending DeleteTablet for 3 replicas of tablet 2909c1748d8840e2afbb0f32593c93d7
I20260929 19:03:22.464107 15743 tablet_service.cc:1514] Processing DeleteTablet for tablet 2909c1748d8840e2afbb0f32593c93d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:22 UTC) from {username='oryx', principal='oryx/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:38064
I20260929 19:03:22.464323 16102 tablet_service.cc:1514] Processing DeleteTablet for tablet 2909c1748d8840e2afbb0f32593c93d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:22 UTC) from {username='oryx', principal='oryx/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:48292
I20260929 19:03:22.464337 15903 tablet_service.cc:1514] Processing DeleteTablet for tablet 2909c1748d8840e2afbb0f32593c93d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:22 UTC) from {username='oryx', principal='oryx/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:46542
I20260929 19:03:22.464509 16216 tablet_replica.cc:330] T 2909c1748d8840e2afbb0f32593c93d7 P b57b14eb0e6a4cb5a5df9a09c73924e7: stopping tablet replica
I20260929 19:03:22.464601 16216 raft_consensus.cc:2229] T 2909c1748d8840e2afbb0f32593c93d7 P b57b14eb0e6a4cb5a5df9a09c73924e7 [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:03:22.464728 16216 raft_consensus.cc:2258] T 2909c1748d8840e2afbb0f32593c93d7 P b57b14eb0e6a4cb5a5df9a09c73924e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:03:22.464875 16214 tablet_replica.cc:330] T 2909c1748d8840e2afbb0f32593c93d7 P cc3c76ee46724a5cb427c2f4ac12ab34: stopping tablet replica
I20260929 19:03:22.464972 16214 raft_consensus.cc:2229] T 2909c1748d8840e2afbb0f32593c93d7 P cc3c76ee46724a5cb427c2f4ac12ab34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:03:22.465061 16214 raft_consensus.cc:2258] T 2909c1748d8840e2afbb0f32593c93d7 P cc3c76ee46724a5cb427c2f4ac12ab34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:03:22.465221 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 14883
I20260929 19:03:22.465919 16216 ts_tablet_manager.cc:1905] T 2909c1748d8840e2afbb0f32593c93d7 P b57b14eb0e6a4cb5a5df9a09c73924e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:03:22.466053 16214 ts_tablet_manager.cc:1905] T 2909c1748d8840e2afbb0f32593c93d7 P cc3c76ee46724a5cb427c2f4ac12ab34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:03:22.467782 16214 ts_tablet_manager.cc:1918] T 2909c1748d8840e2afbb0f32593c93d7 P cc3c76ee46724a5cb427c2f4ac12ab34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:03:22.467903 16216 ts_tablet_manager.cc:1918] T 2909c1748d8840e2afbb0f32593c93d7 P b57b14eb0e6a4cb5a5df9a09c73924e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:03:22.467926 16214 log.cc:1196] T 2909c1748d8840e2afbb0f32593c93d7 P cc3c76ee46724a5cb427c2f4ac12ab34: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-0/wal/wals/2909c1748d8840e2afbb0f32593c93d7
I20260929 19:03:22.467984 16216 log.cc:1196] T 2909c1748d8840e2afbb0f32593c93d7 P b57b14eb0e6a4cb5a5df9a09c73924e7: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-2/wal/wals/2909c1748d8840e2afbb0f32593c93d7
I20260929 19:03:22.468250 16216 ts_tablet_manager.cc:1939] T 2909c1748d8840e2afbb0f32593c93d7 P b57b14eb0e6a4cb5a5df9a09c73924e7: Deleting consensus metadata
I20260929 19:03:22.468405 16214 ts_tablet_manager.cc:1939] T 2909c1748d8840e2afbb0f32593c93d7 P cc3c76ee46724a5cb427c2f4ac12ab34: Deleting consensus metadata
I20260929 19:03:22.468636 16217 tablet_replica.cc:330] T 2909c1748d8840e2afbb0f32593c93d7 P a7a17ab10f8e4693b7d4321931c44da6: stopping tablet replica
I20260929 19:03:22.468739 16217 raft_consensus.cc:2229] T 2909c1748d8840e2afbb0f32593c93d7 P a7a17ab10f8e4693b7d4321931c44da6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:03:22.468835 16217 raft_consensus.cc:2258] T 2909c1748d8840e2afbb0f32593c93d7 P a7a17ab10f8e4693b7d4321931c44da6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:03:22.470767 16217 ts_tablet_manager.cc:1905] T 2909c1748d8840e2afbb0f32593c93d7 P a7a17ab10f8e4693b7d4321931c44da6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 19:03:22.471566 15692 connection.cc:535] server connection from 127.0.0.1:38064 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260929 19:03:22.472369 16057 connection.cc:535] server connection from 127.0.0.1:48292 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 19:03:22.472671 16217 ts_tablet_manager.cc:1918] T 2909c1748d8840e2afbb0f32593c93d7 P a7a17ab10f8e4693b7d4321931c44da6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:03:22.472807 16217 log.cc:1196] T 2909c1748d8840e2afbb0f32593c93d7 P a7a17ab10f8e4693b7d4321931c44da6: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-1/wal/wals/2909c1748d8840e2afbb0f32593c93d7
I20260929 19:03:22.473155 16217 ts_tablet_manager.cc:1939] T 2909c1748d8840e2afbb0f32593c93d7 P a7a17ab10f8e4693b7d4321931c44da6: Deleting consensus metadata
W20260929 19:03:22.473559 15835 connection.cc:441] server connection from 127.0.0.1:46542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46542 (request call id 1) could send its response
I20260929 19:03:22.479363 15576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:22.464696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56958 (local address 127.8.158.125:42397)
0929 19:03:22.464747 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:22.464749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:22.470594 (+ 5845us) server_negotiation.cc:407] Connection header received
0929 19:03:22.470703 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:22.470704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:22.470715 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:22.470743 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:22.471540 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.472283 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.473542 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.473838 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.473884 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:22.475676 (+ 1792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:22.475677 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:22.475678 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:22.475684 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:22.477366 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:22.478471 (+ 1105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:22.478473 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:22.478474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:22.478536 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:22.478755 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:22.478756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:22.478756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:22.478886 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:22.478982 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:22.479183 (+ 201us) server_negotiation.cc:299] Negotiation successful
0929 19:03:22.479262 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 19:03:22.482296 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:22.473679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58234 (local address 127.8.158.124:46133)
0929 19:03:22.473788 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:22.473791 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:22.474181 (+ 390us) server_negotiation.cc:407] Connection header received
0929 19:03:22.474276 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:22.474278 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:22.474290 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:22.474357 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:22.475148 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.475927 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.477051 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.477382 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.477421 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:22.479046 (+ 1625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:22.479048 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:22.479048 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:22.479054 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:22.481040 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:22.481518 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:22.481519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:22.481519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:22.481573 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:22.481759 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:22.481760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:22.481761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:22.481872 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:22.481937 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:22.482116 (+ 179us) server_negotiation.cc:299] Negotiation successful
0929 19:03:22.482188 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":52,"threads_started":1}
I20260929 19:03:22.482585 16222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:22.473341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56972 (local address 127.8.158.125:42397)
0929 19:03:22.473887 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:22.473889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:22.474023 (+ 134us) server_negotiation.cc:407] Connection header received
0929 19:03:22.474047 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:22.474048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:22.474060 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:22.474122 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:22.474936 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.475756 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.476953 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.477295 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.477338 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:22.478885 (+ 1547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:22.478886 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:22.478887 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:22.478893 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:22.481227 (+ 2334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:22.481672 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:22.481673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:22.481674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:22.481731 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:22.481933 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:22.481933 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:22.481934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:22.482057 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:22.482147 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:22.482377 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 19:03:22.482470 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":42,"threads_started":1}
I20260929 19:03:22.486830 16175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:22.464745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58224 (local address 127.8.158.124:46133)
0929 19:03:22.464785 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:22.464787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:22.480038 (+ 15251us) server_negotiation.cc:407] Connection header received
0929 19:03:22.480119 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:22.480120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:22.480128 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:22.480173 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:22.480706 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.481417 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.482419 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:22.482735 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:22.482774 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:22.484024 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:22.484025 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:22.484025 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:22.484032 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:22.485643 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:22.485970 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:22.485971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:22.485972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:22.486043 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:22.486228 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:22.486229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:22.486230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:22.486431 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:22.486534 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:22.486654 (+ 120us) server_negotiation.cc:299] Negotiation successful
0929 19:03:22.486734 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 19:03:23.460254 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:23.459828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:37015 (local address 127.8.158.66:33847)
0929 19:03:23.460137 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:23.460201 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:37015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":60,"threads_started":1}
I20260929 19:03:23.460289 16228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:23.459992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:37015 (local address 127.8.158.67:53603)
0929 19:03:23.460148 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:23.460236 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:37015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":44,"threads_started":1}
W20260929 19:03:23.460711 16169 heartbeater.cc:644] Failed to heartbeat to 127.8.158.126:37015 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.158.126:37015: connect: Connection refused (error 111)
I20260929 19:03:23.460825 16229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:23.460599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:37015 (local address 127.8.158.65:58217)
0929 19:03:23.460688 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:23.460773 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:37015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75,"thread_start_us":47,"threads_started":1}
I20260929 19:03:23.672067 16230 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:03:23.672287 16230 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: false } } peers { permanent_uuid: "9cee7e7f18ff4e888a7b900cc2049cbf" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46133 } attrs { promote: false } }
I20260929 19:03:23.672586 16230 leader_election.cc:288] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b3b7fb74082b41ab8c6bbdb28a50707f (127.8.158.126:37015), 1f3c183506914e0eaa46bcb72ac9c400 (127.8.158.125:42397)
I20260929 19:03:23.672958 16231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:23.672714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:37015 (local address 127.0.0.1:59008)
0929 19:03:23.672857 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:23.672913 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:37015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":53,"threads_started":1}
W20260929 19:03:23.673130 15858 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.158.126:37015: connect: Connection refused (error 111)
I20260929 19:03:23.673543 16231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:23.673447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:37015 (local address 127.0.0.1:59022)
0929 19:03:23.673495 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:23.673520 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:37015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 19:03:23.673664 15858 leader_election.cc:334] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b3b7fb74082b41ab8c6bbdb28a50707f (127.8.158.126:37015): Network error: Client connection negotiation failed: client connection to 127.8.158.126:37015: connect: Connection refused (error 111)
I20260929 19:03:23.675669 16233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:23.672872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56978 (local address 127.8.158.125:42397)
0929 19:03:23.672991 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:23.672993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:23.673004 (+ 11us) server_negotiation.cc:407] Connection header received
0929 19:03:23.673053 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:23.673054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:23.673066 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:23.673127 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:23.673306 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:23.673939 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:23.674948 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:23.675322 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:23.675408 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:23.675419 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:23.675564 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 19:03:23.675605 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":52,"threads_started":1}
I20260929 19:03:23.675676 16232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:23.672742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.125:42397 (local address 127.0.0.1:56978)
0929 19:03:23.672888 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:23.672899 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:23.672987 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:23.673139 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:23.673140 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:23.673142 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:23.673239 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:23.673242 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:23.674023 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:23.674024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:23.674856 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:23.674858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:23.675408 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:23.675408 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:23.675489 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:23.675492 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:03:23.675560 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 19:03:23.675602 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":52,"threads_started":1}
I20260929 19:03:23.676036 15535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9cee7e7f18ff4e888a7b900cc2049cbf" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "1f3c183506914e0eaa46bcb72ac9c400" is_pre_election: true
I20260929 19:03:23.676313 15862 leader_election.cc:302] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9cee7e7f18ff4e888a7b900cc2049cbf; no voters: 1f3c183506914e0eaa46bcb72ac9c400, b3b7fb74082b41ab8c6bbdb28a50707f
I20260929 19:03:23.676424 16230 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 19:03:23.964758 16235 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b3b7fb74082b41ab8c6bbdb28a50707f)
I20260929 19:03:23.965286 16235 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: false } } peers { permanent_uuid: "9cee7e7f18ff4e888a7b900cc2049cbf" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46133 } attrs { promote: false } }
I20260929 19:03:23.966101 16235 leader_election.cc:288] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b3b7fb74082b41ab8c6bbdb28a50707f (127.8.158.126:37015), 9cee7e7f18ff4e888a7b900cc2049cbf (127.8.158.124:46133)
I20260929 19:03:23.966576 16237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:23.966192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:37015 (local address 127.0.0.1:59078)
0929 19:03:23.966398 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:23.966503 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:37015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":57,"threads_started":1}
W20260929 19:03:23.966789 15509 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.158.126:37015: connect: Connection refused (error 111)
I20260929 19:03:23.967515 16237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:23.967410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:37015 (local address 127.0.0.1:59092)
0929 19:03:23.967458 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:23.967489 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:37015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 19:03:23.967626 15509 leader_election.cc:334] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b3b7fb74082b41ab8c6bbdb28a50707f (127.8.158.126:37015): Network error: Client connection negotiation failed: client connection to 127.8.158.126:37015: connect: Connection refused (error 111)
I20260929 19:03:23.970096 16236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:23.966264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.124:46133 (local address 127.0.0.1:58248)
0929 19:03:23.966416 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:23.966430 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:23.966492 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:23.966729 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:23.966730 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:23.966741 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:23.966801 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:23.966803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:23.967677 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:23.967678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:23.968912 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:23.968915 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:23.969580 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:23.969581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:23.969686 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:23.969690 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:03:23.969752 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 19:03:23.969801 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":95,"threads_started":1}
I20260929 19:03:23.970389 16238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:23.966356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58248 (local address 127.8.158.124:46133)
0929 19:03:23.966509 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:23.966511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:23.966520 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:03:23.966566 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:23.966566 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:23.966578 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:23.966657 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:23.966911 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:23.967598 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:23.968997 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:23.969462 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:23.969528 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:23.969539 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:23.970200 (+ 661us) server_negotiation.cc:299] Negotiation successful
0929 19:03:23.970257 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
I20260929 19:03:23.970608 15998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1f3c183506914e0eaa46bcb72ac9c400" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9cee7e7f18ff4e888a7b900cc2049cbf" is_pre_election: true
I20260929 19:03:23.970749 15998 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f3c183506914e0eaa46bcb72ac9c400 in term 1.
I20260929 19:03:23.970940 15510 leader_election.cc:302] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f3c183506914e0eaa46bcb72ac9c400, 9cee7e7f18ff4e888a7b900cc2049cbf; no voters: b3b7fb74082b41ab8c6bbdb28a50707f
I20260929 19:03:23.971074 16235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 19:03:23.971122 16235 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b3b7fb74082b41ab8c6bbdb28a50707f)
I20260929 19:03:23.971144 16235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [term 1 FOLLOWER]: Advancing to term 2
I20260929 19:03:23.972149 16235 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: false } } peers { permanent_uuid: "9cee7e7f18ff4e888a7b900cc2049cbf" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46133 } attrs { promote: false } }
I20260929 19:03:23.972321 16235 leader_election.cc:288] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [CANDIDATE]: Term 2 election: Requested vote from peers b3b7fb74082b41ab8c6bbdb28a50707f (127.8.158.126:37015), 9cee7e7f18ff4e888a7b900cc2049cbf (127.8.158.124:46133)
I20260929 19:03:23.972527 16236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:23.972404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:37015 (local address 127.0.0.1:59106)
0929 19:03:23.972452 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:23.972481 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:37015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 19:03:23.972533 15998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1f3c183506914e0eaa46bcb72ac9c400" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9cee7e7f18ff4e888a7b900cc2049cbf"
I20260929 19:03:23.972610 15998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf [term 1 FOLLOWER]: Advancing to term 2
I20260929 19:03:23.973474 15998 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f3c183506914e0eaa46bcb72ac9c400 in term 2.
I20260929 19:03:23.973670 15510 leader_election.cc:302] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f3c183506914e0eaa46bcb72ac9c400, 9cee7e7f18ff4e888a7b900cc2049cbf; no voters:
I20260929 19:03:23.973799 16235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 19:03:23.973888 16236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:23.973793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:37015 (local address 127.0.0.1:59110)
0929 19:03:23.973841 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:23.973865 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:37015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260929 19:03:23.973917 16235 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [term 2 LEADER]: Becoming Leader. State: Replica: 1f3c183506914e0eaa46bcb72ac9c400, State: Running, Role: LEADER
W20260929 19:03:23.973951 15509 leader_election.cc:334] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b3b7fb74082b41ab8c6bbdb28a50707f (127.8.158.126:37015): Network error: Client connection negotiation failed: client connection to 127.8.158.126:37015: connect: Connection refused (error 111)
I20260929 19:03:23.973979 16235 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: false } } peers { permanent_uuid: "9cee7e7f18ff4e888a7b900cc2049cbf" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46133 } attrs { promote: false } }
I20260929 19:03:23.974303 16240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f3c183506914e0eaa46bcb72ac9c400. Latest consensus state: current_term: 2 leader_uuid: "1f3c183506914e0eaa46bcb72ac9c400" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: false } } peers { permanent_uuid: "9cee7e7f18ff4e888a7b900cc2049cbf" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46133 } attrs { promote: false } } }
I20260929 19:03:23.974406 16240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:23.974594 16242 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:03:23.975072 16242 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:03:23.975168 16242 catalog_manager.cc:1259] Loaded cluster ID: 7c8d5b9d44a245eba27c93910892b2b7
I20260929 19:03:23.975203 16242 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:03:23.975430 16242 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:03:23.975674 16242 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400: Loaded TSK: 0
I20260929 19:03:23.975862 16242 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:03:24.019328 15524 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56972:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:03:24.020853 15524 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:03:24.021827 15998 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf [term 2 FOLLOWER]: Refusing update from remote peer 1f3c183506914e0eaa46bcb72ac9c400: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260929 19:03:24.022111 16236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:24.021735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:37015 (local address 127.0.0.1:59128)
0929 19:03:24.022024 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:24.022057 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:37015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270}
I20260929 19:03:24.022055 16240 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cee7e7f18ff4e888a7b900cc2049cbf" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 8, Time since last communication: 0.000s
I20260929 19:03:24.022884 16236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:24.022356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:37015 (local address 127.0.0.1:59138)
0929 19:03:24.022824 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:24.022856 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:37015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449}
I20260929 19:03:24.022833 16230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f3c183506914e0eaa46bcb72ac9c400. Latest consensus state: current_term: 2 leader_uuid: "1f3c183506914e0eaa46bcb72ac9c400" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: false } } peers { permanent_uuid: "9cee7e7f18ff4e888a7b900cc2049cbf" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46133 } attrs { promote: false } } }
I20260929 19:03:24.022941 16230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf [sys.catalog]: This master's current role is: FOLLOWER
W20260929 19:03:24.023005 15509 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 -> Peer b3b7fb74082b41ab8c6bbdb28a50707f (127.8.158.126:37015): Couldn't send request to peer b3b7fb74082b41ab8c6bbdb28a50707f. Status: Network error: Client connection negotiation failed: client connection to 127.8.158.126:37015: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 19:03:24.023325 16240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "1f3c183506914e0eaa46bcb72ac9c400" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: false } } peers { permanent_uuid: "9cee7e7f18ff4e888a7b900cc2049cbf" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46133 } attrs { promote: false } } }
I20260929 19:03:24.023416 16240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400 [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:24.023408 16230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "1f3c183506914e0eaa46bcb72ac9c400" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b3b7fb74082b41ab8c6bbdb28a50707f" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37015 } } peers { permanent_uuid: "1f3c183506914e0eaa46bcb72ac9c400" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 42397 } attrs { promote: false } } peers { permanent_uuid: "9cee7e7f18ff4e888a7b900cc2049cbf" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46133 } attrs { promote: false } } }
I20260929 19:03:24.023481 16230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cee7e7f18ff4e888a7b900cc2049cbf [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:03:24.028393 16236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:24.025040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.66:42523 (local address 127.0.0.1:46550)
0929 19:03:24.025131 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:24.025142 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:24.025182 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:24.025441 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:24.025442 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:24.025444 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:24.025495 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:24.025497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:24.026157 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:24.026158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:24.027388 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:24.027390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:24.028140 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:24.028141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:24.028198 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:24.028203 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 19:03:24.028265 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 19:03:24.028305 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":54}
I20260929 19:03:24.028548 16237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:24.025128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:43095 (local address 127.0.0.1:38080)
0929 19:03:24.025251 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:24.025261 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:24.025313 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:24.025496 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:24.025497 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:24.025500 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:24.025557 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:24.025558 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:24.026392 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:24.026393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:24.027559 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:24.027561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:24.028307 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:24.028308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:24.028374 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:24.028378 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:03:24.028435 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 19:03:24.028471 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260929 19:03:24.028625 16250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:24.025154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46550 (local address 127.8.158.66:42523)
0929 19:03:24.025321 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:24.025323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:24.025331 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:03:24.025357 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:24.025358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:24.025370 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:24.025441 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:24.025585 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:24.026083 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:24.027479 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:24.028037 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:24.028100 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:24.028111 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:24.028512 (+ 401us) server_negotiation.cc:299] Negotiation successful
0929 19:03:24.028560 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":58,"threads_started":1}
I20260929 19:03:24.028628 16249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:24.025158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38080 (local address 127.8.158.65:43095)
0929 19:03:24.025256 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:24.025258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:24.025362 (+ 104us) server_negotiation.cc:407] Connection header received
0929 19:03:24.025390 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:24.025390 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:24.025401 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:24.025489 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:24.025666 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:24.026290 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:24.027640 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:24.028220 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:24.028284 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:24.028294 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:24.028454 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 19:03:24.028568 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":51,"threads_started":1}
I20260929 19:03:24.028821 15903 tablet_service.cc:1467] Processing CreateTablet for tablet a7b3493272f64b74a5e2e2bea5319d7b (DEFAULT_TABLE table=test-table [id=4046b62694504cf2b922198913db3c56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:24.028947 15903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7b3493272f64b74a5e2e2bea5319d7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:24.029374 16251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:24.025312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.67:38403 (local address 127.0.0.1:48322)
0929 19:03:24.025492 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:24.025501 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:24.025540 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:24.026293 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:24.026293 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:24.026296 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:24.026387 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:24.026389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:24.027070 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:24.027071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:24.027935 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:24.027937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:24.029187 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:24.029187 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:24.029240 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:24.029244 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:03:24.029290 (+ 46us) client_negotiation.cc:240] Negotiation successful
0929 19:03:24.029316 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":64,"threads_started":1}
I20260929 19:03:24.029408 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:24.025395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48322 (local address 127.8.158.67:38403)
0929 19:03:24.026178 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:24.026179 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:24.026185 (+ 6us) server_negotiation.cc:407] Connection header received
0929 19:03:24.026205 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:24.026206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:24.026214 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:24.026290 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:24.026449 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:24.026969 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:24.028014 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:24.029111 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:24.029195 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:24.029206 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:24.029311 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 19:03:24.029348 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":766,"thread_start_us":51,"threads_started":1}
I20260929 19:03:24.029613 16102 tablet_service.cc:1467] Processing CreateTablet for tablet a7b3493272f64b74a5e2e2bea5319d7b (DEFAULT_TABLE table=test-table [id=4046b62694504cf2b922198913db3c56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:24.029750 16102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7b3493272f64b74a5e2e2bea5319d7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:24.030423 15743 tablet_service.cc:1467] Processing CreateTablet for tablet a7b3493272f64b74a5e2e2bea5319d7b (DEFAULT_TABLE table=test-table [id=4046b62694504cf2b922198913db3c56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:24.030476 16253 tablet_bootstrap.cc:492] T a7b3493272f64b74a5e2e2bea5319d7b P a7a17ab10f8e4693b7d4321931c44da6: Bootstrap starting.
I20260929 19:03:24.030534 15743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7b3493272f64b74a5e2e2bea5319d7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:24.030838 16253 tablet_bootstrap.cc:654] T a7b3493272f64b74a5e2e2bea5319d7b P a7a17ab10f8e4693b7d4321931c44da6: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:24.031123 16254 tablet_bootstrap.cc:492] T a7b3493272f64b74a5e2e2bea5319d7b P b57b14eb0e6a4cb5a5df9a09c73924e7: Bootstrap starting.
I20260929 19:03:24.031469 16254 tablet_bootstrap.cc:654] T a7b3493272f64b74a5e2e2bea5319d7b P b57b14eb0e6a4cb5a5df9a09c73924e7: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:24.031533 16253 tablet_bootstrap.cc:492] T a7b3493272f64b74a5e2e2bea5319d7b P a7a17ab10f8e4693b7d4321931c44da6: No bootstrap required, opened a new log
I20260929 19:03:24.031571 16253 ts_tablet_manager.cc:1397] T a7b3493272f64b74a5e2e2bea5319d7b P a7a17ab10f8e4693b7d4321931c44da6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:24.031675 16253 raft_consensus.cc:348] T a7b3493272f64b74a5e2e2bea5319d7b P a7a17ab10f8e4693b7d4321931c44da6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7a17ab10f8e4693b7d4321931c44da6" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 42523 } } peers { permanent_uuid: "cc3c76ee46724a5cb427c2f4ac12ab34" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 43095 } } peers { permanent_uuid: "b57b14eb0e6a4cb5a5df9a09c73924e7" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38403 } }
I20260929 19:03:24.031764 16253 raft_consensus.cc:374] T a7b3493272f64b74a5e2e2bea5319d7b P a7a17ab10f8e4693b7d4321931c44da6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:24.031803 16253 raft_consensus.cc:729] T a7b3493272f64b74a5e2e2bea5319d7b P a7a17ab10f8e4693b7d4321931c44da6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7a17ab10f8e4693b7d4321931c44da6, State: Initialized, Role: FOLLOWER
I20260929 19:03:24.031869 16253 consensus_queue.cc:260] T a7b3493272f64b74a5e2e2bea5319d7b P a7a17ab10f8e4693b7d4321931c44da6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7a17ab10f8e4693b7d4321931c44da6" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 42523 } } peers { permanent_uuid: "cc3c76ee46724a5cb427c2f4ac12ab34" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 43095 } } peers { permanent_uuid: "b57b14eb0e6a4cb5a5df9a09c73924e7" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38403 } }
I20260929 19:03:24.031991 16257 tablet_bootstrap.cc:492] T a7b3493272f64b74a5e2e2bea5319d7b P cc3c76ee46724a5cb427c2f4ac12ab34: Bootstrap starting.
I20260929 19:03:24.032025 16253 ts_tablet_manager.cc:1428] T a7b3493272f64b74a5e2e2bea5319d7b P a7a17ab10f8e4693b7d4321931c44da6: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 19:03:24.032064 16254 tablet_bootstrap.cc:492] T a7b3493272f64b74a5e2e2bea5319d7b P b57b14eb0e6a4cb5a5df9a09c73924e7: No bootstrap required, opened a new log
I20260929 19:03:24.032111 16254 ts_tablet_manager.cc:1397] T a7b3493272f64b74a5e2e2bea5319d7b P b57b14eb0e6a4cb5a5df9a09c73924e7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:24.032218 16254 raft_consensus.cc:348] T a7b3493272f64b74a5e2e2bea5319d7b P b57b14eb0e6a4cb5a5df9a09c73924e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7a17ab10f8e4693b7d4321931c44da6" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 42523 } } peers { permanent_uuid: "cc3c76ee46724a5cb427c2f4ac12ab34" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 43095 } } peers { permanent_uuid: "b57b14eb0e6a4cb5a5df9a09c73924e7" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38403 } }
I20260929 19:03:24.032296 16254 raft_consensus.cc:374] T a7b3493272f64b74a5e2e2bea5319d7b P b57b14eb0e6a4cb5a5df9a09c73924e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:24.032322 16254 raft_consensus.cc:729] T a7b3493272f64b74a5e2e2bea5319d7b P b57b14eb0e6a4cb5a5df9a09c73924e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b57b14eb0e6a4cb5a5df9a09c73924e7, State: Initialized, Role: FOLLOWER
I20260929 19:03:24.032377 16254 consensus_queue.cc:260] T a7b3493272f64b74a5e2e2bea5319d7b P b57b14eb0e6a4cb5a5df9a09c73924e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7a17ab10f8e4693b7d4321931c44da6" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 42523 } } peers { permanent_uuid: "cc3c76ee46724a5cb427c2f4ac12ab34" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 43095 } } peers { permanent_uuid: "b57b14eb0e6a4cb5a5df9a09c73924e7" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38403 } }
I20260929 19:03:24.032507 16254 ts_tablet_manager.cc:1428] T a7b3493272f64b74a5e2e2bea5319d7b P b57b14eb0e6a4cb5a5df9a09c73924e7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 19:03:24.032958 16257 tablet_bootstrap.cc:654] T a7b3493272f64b74a5e2e2bea5319d7b P cc3c76ee46724a5cb427c2f4ac12ab34: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:24.032991 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:24.032806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:37015 (local address 127.8.158.67:32975)
0929 19:03:24.032922 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:24.032959 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:37015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":76,"threads_started":1}
I20260929 19:03:24.033139 15971 heartbeater.cc:503] Master 127.8.158.125:42397 was elected leader, sending a full tablet report...
I20260929 19:03:24.033145 16259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:24.032196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:37015 (local address 127.8.158.66:48647)
0929 19:03:24.033061 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:24.033104 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:37015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":82,"threads_started":1}
I20260929 19:03:24.033514 16170 heartbeater.cc:503] Master 127.8.158.125:42397 was elected leader, sending a full tablet report...
I20260929 19:03:24.033849 16259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:24.033716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:37015 (local address 127.8.158.66:36409)
0929 19:03:24.033799 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:24.033826 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:37015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 19:03:24.034056 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:24.033629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:37015 (local address 127.8.158.67:51025)
0929 19:03:24.034007 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:24.034038 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:37015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365}
I20260929 19:03:24.034488 16257 tablet_bootstrap.cc:492] T a7b3493272f64b74a5e2e2bea5319d7b P cc3c76ee46724a5cb427c2f4ac12ab34: No bootstrap required, opened a new log
I20260929 19:03:24.034540 16257 ts_tablet_manager.cc:1397] T a7b3493272f64b74a5e2e2bea5319d7b P cc3c76ee46724a5cb427c2f4ac12ab34: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 19:03:24.034657 16257 raft_consensus.cc:348] T a7b3493272f64b74a5e2e2bea5319d7b P cc3c76ee46724a5cb427c2f4ac12ab34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7a17ab10f8e4693b7d4321931c44da6" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 42523 } } peers { permanent_uuid: "cc3c76ee46724a5cb427c2f4ac12ab34" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 43095 } } peers { permanent_uuid: "b57b14eb0e6a4cb5a5df9a09c73924e7" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38403 } }
I20260929 19:03:24.034725 16257 raft_consensus.cc:374] T a7b3493272f64b74a5e2e2bea5319d7b P cc3c76ee46724a5cb427c2f4ac12ab34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:24.034744 16257 raft_consensus.cc:729] T a7b3493272f64b74a5e2e2bea5319d7b P cc3c76ee46724a5cb427c2f4ac12ab34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc3c76ee46724a5cb427c2f4ac12ab34, State: Initialized, Role: FOLLOWER
I20260929 19:03:24.034792 16257 consensus_queue.cc:260] T a7b3493272f64b74a5e2e2bea5319d7b P cc3c76ee46724a5cb427c2f4ac12ab34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7a17ab10f8e4693b7d4321931c44da6" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 42523 } } peers { permanent_uuid: "cc3c76ee46724a5cb427c2f4ac12ab34" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 43095 } } peers { permanent_uuid: "b57b14eb0e6a4cb5a5df9a09c73924e7" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38403 } }
I20260929 19:03:24.034901 16257 ts_tablet_manager.cc:1428] T a7b3493272f64b74a5e2e2bea5319d7b P cc3c76ee46724a5cb427c2f4ac12ab34: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 19:03:24.035254 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:24.035062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:37015 (local address 127.8.158.65:40341)
0929 19:03:24.035169 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:24.035211 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:37015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":75,"threads_started":1}
I20260929 19:03:24.035338 15811 heartbeater.cc:503] Master 127.8.158.125:42397 was elected leader, sending a full tablet report...
I20260929 19:03:24.035501 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:24.035430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:37015 (local address 127.8.158.65:50563)
0929 19:03:24.035458 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:24.035480 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:37015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260929 19:03:24.060060 16171 tablet.cc:2369] T a7b3493272f64b74a5e2e2bea5319d7b P b57b14eb0e6a4cb5a5df9a09c73924e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:03:24.069307 16258 raft_consensus.cc:482] T a7b3493272f64b74a5e2e2bea5319d7b P a7a17ab10f8e4693b7d4321931c44da6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:03:24.069375 16258 raft_consensus.cc:504] T a7b3493272f64b74a5e2e2bea5319d7b P a7a17ab10f8e4693b7d4321931c44da6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7a17ab10f8e4693b7d4321931c44da6" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 42523 } } peers { permanent_uuid: "cc3c76ee46724a5cb427c2f4ac12ab34" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 43095 } } peers { permanent_uuid: "b57b14eb0e6a4cb5a5df9a09c73924e7" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38403 } }
I20260929 19:03:24.069738 16258 leader_election.cc:288] T a7b3493272f64b74a5e2e2bea5319d7b P a7a17ab10f8e4693b7d4321931c44da6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc3c76ee46724a5cb427c2f4ac12ab34 (127.8.158.65:43095), b57b14eb0e6a4cb5a5df9a09c73924e7 (127.8.158.67:38403)
I20260929 19:03:24.073459 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:24.070000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:53129 (local address 127.8.158.67:38403)
0929 19:03:24.070042 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:24.070044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:24.070117 (+ 73us) server_negotiation.cc:407] Connection header received
0929 19:03:24.070147 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:24.070148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:24.070155 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:24.070239 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:24.070347 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:24.071040 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:24.072470 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:24.073091 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:24.073165 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:24.073178 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:24.073356 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 19:03:24.073396 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 19:03:24.073459 16249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:24.069893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:58749 (local address 127.8.158.65:43095)
0929 19:03:24.069932 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:24.069933 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:24.069943 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:03:24.069997 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:24.069998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:24.070007 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:24.070066 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:24.070236 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:24.071017 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:24.072481 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:24.073132 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:24.073193 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:24.073201 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:24.073356 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 19:03:24.073394 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 19:03:24.073483 16259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:24.069822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:43095 (local address 127.8.158.66:58749)
0929 19:03:24.069869 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:24.069881 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:24.069941 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:24.070077 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:24.070078 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:24.070080 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:24.070153 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:24.070155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:24.071152 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:24.071154 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:24.072362 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:24.072364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:24.073189 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:24.073190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:24.073269 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:24.073272 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:03:24.073341 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 19:03:24.073389 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":43}
I20260929 19:03:24.073768 15763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7b3493272f64b74a5e2e2bea5319d7b" candidate_uuid: "a7a17ab10f8e4693b7d4321931c44da6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc3c76ee46724a5cb427c2f4ac12ab34" is_pre_election: true
I20260929 19:03:24.073856 15763 raft_consensus.cc:2454] T a7b3493272f64b74a5e2e2bea5319d7b P cc3c76ee46724a5cb427c2f4ac12ab34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7a17ab10f8e4693b7d4321931c44da6 in term 0.
I20260929 19:03:24.073493 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:24.069883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.67:38403 (local address 127.8.158.66:53129)
0929 19:03:24.070039 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:24.070049 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:24.070105 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:24.070248 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:24.070248 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:24.070249 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:24.070277 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:24.070278 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:24.071153 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:24.071154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:24.072362 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:24.072364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:24.073178 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:24.073180 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:24.073269 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:24.073273 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:03:24.073334 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 19:03:24.073389 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":90,"thread_start_us":90,"threads_started":1}
I20260929 19:03:24.074183 15837 leader_election.cc:302] T a7b3493272f64b74a5e2e2bea5319d7b P a7a17ab10f8e4693b7d4321931c44da6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7a17ab10f8e4693b7d4321931c44da6, cc3c76ee46724a5cb427c2f4ac12ab34; no voters:
I20260929 19:03:24.074196 16122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7b3493272f64b74a5e2e2bea5319d7b" candidate_uuid: "a7a17ab10f8e4693b7d4321931c44da6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b57b14eb0e6a4cb5a5df9a09c73924e7" is_pre_election: true
I20260929 19:03:24.074333 16258 raft_consensus.cc:2781] T a7b3493272f64b74a5e2e2bea5319d7b P a7a17ab10f8e4693b7d4321931c44da6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:03:24.074378 16258 raft_consensus.cc:482] T a7b3493272f64b74a5e2e2bea5319d7b P a7a17ab10f8e4693b7d4321931c44da6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:03:24.074366 16122 raft_consensus.cc:2454] T a7b3493272f64b74a5e2e2bea5319d7b P b57b14eb0e6a4cb5a5df9a09c73924e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7a17ab10f8e4693b7d4321931c44da6 in term 0.
I20260929 19:03:24.074409 16258 raft_consensus.cc:3037] T a7b3493272f64b74a5e2e2bea5319d7b P a7a17ab10f8e4693b7d4321931c44da6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:24.075111 16258 raft_consensus.cc:504] T a7b3493272f64b74a5e2e2bea5319d7b P a7a17ab10f8e4693b7d4321931c44da6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7a17ab10f8e4693b7d4321931c44da6" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 42523 } } peers { permanent_uuid: "cc3c76ee46724a5cb427c2f4ac12ab34" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 43095 } } peers { permanent_uuid: "b57b14eb0e6a4cb5a5df9a09c73924e7" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38403 } }
I20260929 19:03:24.075263 16258 leader_election.cc:288] T a7b3493272f64b74a5e2e2bea5319d7b P a7a17ab10f8e4693b7d4321931c44da6 [CANDIDATE]: Term 1 election: Requested vote from peers cc3c76ee46724a5cb427c2f4ac12ab34 (127.8.158.65:43095), b57b14eb0e6a4cb5a5df9a09c73924e7 (127.8.158.67:38403)
I20260929 19:03:24.075376 15763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7b3493272f64b74a5e2e2bea5319d7b" candidate_uuid: "a7a17ab10f8e4693b7d4321931c44da6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc3c76ee46724a5cb427c2f4ac12ab34"
I20260929 19:03:24.075438 15763 raft_consensus.cc:3037] T a7b3493272f64b74a5e2e2bea5319d7b P cc3c76ee46724a5cb427c2f4ac12ab34 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:24.075490 16122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7b3493272f64b74a5e2e2bea5319d7b" candidate_uuid: "a7a17ab10f8e4693b7d4321931c44da6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b57b14eb0e6a4cb5a5df9a09c73924e7"
I20260929 19:03:24.075558 16122 raft_consensus.cc:3037] T a7b3493272f64b74a5e2e2bea5319d7b P b57b14eb0e6a4cb5a5df9a09c73924e7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:24.076016 15763 raft_consensus.cc:2454] T a7b3493272f64b74a5e2e2bea5319d7b P cc3c76ee46724a5cb427c2f4ac12ab34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7a17ab10f8e4693b7d4321931c44da6 in term 1.
I20260929 19:03:24.076205 15837 leader_election.cc:302] T a7b3493272f64b74a5e2e2bea5319d7b P a7a17ab10f8e4693b7d4321931c44da6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7a17ab10f8e4693b7d4321931c44da6, cc3c76ee46724a5cb427c2f4ac12ab34; no voters:
I20260929 19:03:24.076290 16258 raft_consensus.cc:2781] T a7b3493272f64b74a5e2e2bea5319d7b P a7a17ab10f8e4693b7d4321931c44da6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:03:24.076309 16122 raft_consensus.cc:2454] T a7b3493272f64b74a5e2e2bea5319d7b P b57b14eb0e6a4cb5a5df9a09c73924e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7a17ab10f8e4693b7d4321931c44da6 in term 1.
I20260929 19:03:24.076458 16258 raft_consensus.cc:686] T a7b3493272f64b74a5e2e2bea5319d7b P a7a17ab10f8e4693b7d4321931c44da6 [term 1 LEADER]: Becoming Leader. State: Replica: a7a17ab10f8e4693b7d4321931c44da6, State: Running, Role: LEADER
I20260929 19:03:24.076629 16258 consensus_queue.cc:237] T a7b3493272f64b74a5e2e2bea5319d7b P a7a17ab10f8e4693b7d4321931c44da6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7a17ab10f8e4693b7d4321931c44da6" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 42523 } } peers { permanent_uuid: "cc3c76ee46724a5cb427c2f4ac12ab34" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 43095 } } peers { permanent_uuid: "b57b14eb0e6a4cb5a5df9a09c73924e7" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38403 } }
I20260929 19:03:24.076803 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:24.076670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:37015 (local address 127.8.158.66:58259)
0929 19:03:24.076735 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:24.076770 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:37015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 19:03:24.077309 15525 catalog_manager.cc:5495] T a7b3493272f64b74a5e2e2bea5319d7b P a7a17ab10f8e4693b7d4321931c44da6 reported cstate change: term changed from 0 to 1, leader changed from <none> to a7a17ab10f8e4693b7d4321931c44da6 (127.8.158.66). New cstate: current_term: 1 leader_uuid: "a7a17ab10f8e4693b7d4321931c44da6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7a17ab10f8e4693b7d4321931c44da6" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 42523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc3c76ee46724a5cb427c2f4ac12ab34" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 43095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b57b14eb0e6a4cb5a5df9a09c73924e7" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38403 } health_report { overall_health: UNKNOWN } } }
I20260929 19:03:24.085330 16250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:24.083330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46562 (local address 127.8.158.66:42523)
0929 19:03:24.083383 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:24.083384 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:24.083393 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:03:24.083425 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:24.083426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:24.083439 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:24.083499 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:03:24.083583 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:24.084004 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:24.084662 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:24.084926 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:24.084980 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:24.085094 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:03:24.085186 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:03:24.085239 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:24.085272 (+ 33us) server_negotiation.cc:299] Negotiation successful
0929 19:03:24.085287 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 19:03:24.086274 15763 raft_consensus.cc:1261] T a7b3493272f64b74a5e2e2bea5319d7b P cc3c76ee46724a5cb427c2f4ac12ab34 [term 1 FOLLOWER]: Refusing update from remote peer a7a17ab10f8e4693b7d4321931c44da6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:03:24.086303 16122 raft_consensus.cc:1261] T a7b3493272f64b74a5e2e2bea5319d7b P b57b14eb0e6a4cb5a5df9a09c73924e7 [term 1 FOLLOWER]: Refusing update from remote peer a7a17ab10f8e4693b7d4321931c44da6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:03:24.086529 16266 consensus_queue.cc:1035] T a7b3493272f64b74a5e2e2bea5319d7b P a7a17ab10f8e4693b7d4321931c44da6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3c76ee46724a5cb427c2f4ac12ab34" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 43095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:03:24.086649 16258 consensus_queue.cc:1035] T a7b3493272f64b74a5e2e2bea5319d7b P a7a17ab10f8e4693b7d4321931c44da6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b57b14eb0e6a4cb5a5df9a09c73924e7" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:03:24.087858 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:24.087607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:37015 (local address 127.8.158.65:60901)
0929 19:03:24.087799 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:24.087829 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:37015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176}
I20260929 19:03:24.087936 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:24.087843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:37015 (local address 127.8.158.67:39103)
0929 19:03:24.087881 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:24.087911 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:37015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260929 19:03:24.088500 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:24.088415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:37015 (local address 127.8.158.66:37973)
0929 19:03:24.088445 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:24.088475 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:37015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260929 19:03:24.089000 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:24.088910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:37015 (local address 127.8.158.66:38719)
0929 19:03:24.088952 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:24.088973 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:37015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 19:03:24.089176 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:24.089100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:37015 (local address 127.8.158.67:38727)
0929 19:03:24.089133 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:24.089153 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:37015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260929 19:03:24.089591 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:24.089511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:37015 (local address 127.8.158.65:34587)
0929 19:03:24.089542 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:24.089564 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:37015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260929 19:03:24.091626 15525 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:03:24.091714 15525 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:03:24.092890 15525 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1f3c183506914e0eaa46bcb72ac9c400: Sending DeleteTablet for 3 replicas of tablet a7b3493272f64b74a5e2e2bea5319d7b
I20260929 19:03:24.093149 15903 tablet_service.cc:1514] Processing DeleteTablet for tablet a7b3493272f64b74a5e2e2bea5319d7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:24 UTC) from {username='oryx', principal='oryx/127.8.158.125@KRBTEST.COM'} at 127.0.0.1:46550
I20260929 19:03:24.093146 15743 tablet_service.cc:1514] Processing DeleteTablet for tablet a7b3493272f64b74a5e2e2bea5319d7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:24 UTC) from {username='oryx', principal='oryx/127.8.158.125@KRBTEST.COM'} at 127.0.0.1:38080
I20260929 19:03:24.093179 16102 tablet_service.cc:1514] Processing DeleteTablet for tablet a7b3493272f64b74a5e2e2bea5319d7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:24 UTC) from {username='oryx', principal='oryx/127.8.158.125@KRBTEST.COM'} at 127.0.0.1:48322
I20260929 19:03:24.093394 16280 tablet_replica.cc:330] T a7b3493272f64b74a5e2e2bea5319d7b P b57b14eb0e6a4cb5a5df9a09c73924e7: stopping tablet replica
I20260929 19:03:24.093451 16280 raft_consensus.cc:2229] T a7b3493272f64b74a5e2e2bea5319d7b P b57b14eb0e6a4cb5a5df9a09c73924e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:03:24.093489 16280 raft_consensus.cc:2258] T a7b3493272f64b74a5e2e2bea5319d7b P b57b14eb0e6a4cb5a5df9a09c73924e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:03:24.093587 16278 tablet_replica.cc:330] T a7b3493272f64b74a5e2e2bea5319d7b P a7a17ab10f8e4693b7d4321931c44da6: stopping tablet replica
I20260929 19:03:24.093678 16278 raft_consensus.cc:2229] T a7b3493272f64b74a5e2e2bea5319d7b P a7a17ab10f8e4693b7d4321931c44da6 [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:03:24.093709 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 15673
I20260929 19:03:24.095198 16280 ts_tablet_manager.cc:1905] T a7b3493272f64b74a5e2e2bea5319d7b P b57b14eb0e6a4cb5a5df9a09c73924e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:03:24.095690 16278 raft_consensus.cc:2258] T a7b3493272f64b74a5e2e2bea5319d7b P a7a17ab10f8e4693b7d4321931c44da6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:03:24.096025 16278 ts_tablet_manager.cc:1905] T a7b3493272f64b74a5e2e2bea5319d7b P a7a17ab10f8e4693b7d4321931c44da6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:03:24.096845 16280 ts_tablet_manager.cc:1918] T a7b3493272f64b74a5e2e2bea5319d7b P b57b14eb0e6a4cb5a5df9a09c73924e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:03:24.096908 16280 log.cc:1196] T a7b3493272f64b74a5e2e2bea5319d7b P b57b14eb0e6a4cb5a5df9a09c73924e7: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-2/wal/wals/a7b3493272f64b74a5e2e2bea5319d7b
I20260929 19:03:24.097144 16280 ts_tablet_manager.cc:1939] T a7b3493272f64b74a5e2e2bea5319d7b P b57b14eb0e6a4cb5a5df9a09c73924e7: Deleting consensus metadata
I20260929 19:03:24.097582 15509 catalog_manager.cc:4841] TS b57b14eb0e6a4cb5a5df9a09c73924e7 (127.8.158.67:38403): tablet a7b3493272f64b74a5e2e2bea5319d7b (table test-table [id=4046b62694504cf2b922198913db3c56]) successfully deleted
I20260929 19:03:24.097608 16278 ts_tablet_manager.cc:1918] T a7b3493272f64b74a5e2e2bea5319d7b P a7a17ab10f8e4693b7d4321931c44da6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:03:24.097653 16278 log.cc:1196] T a7b3493272f64b74a5e2e2bea5319d7b P a7a17ab10f8e4693b7d4321931c44da6: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708573736807-8825-0/minicluster-data/ts-1/wal/wals/a7b3493272f64b74a5e2e2bea5319d7b
I20260929 19:03:24.097921 16278 ts_tablet_manager.cc:1939] T a7b3493272f64b74a5e2e2bea5319d7b P a7a17ab10f8e4693b7d4321931c44da6: Deleting consensus metadata
I20260929 19:03:24.098233 15510 catalog_manager.cc:4841] TS a7a17ab10f8e4693b7d4321931c44da6 (127.8.158.66:42523): tablet a7b3493272f64b74a5e2e2bea5319d7b (table test-table [id=4046b62694504cf2b922198913db3c56]) successfully deleted
I20260929 19:03:24.101157 16251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:24.101055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:43095 (local address 127.0.0.1:38088)
0929 19:03:24.101096 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:24.101126 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.65:43095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260929 19:03:24.101313 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 15816
W20260929 19:03:24.101294 15511 catalog_manager.cc:4584] TS cc3c76ee46724a5cb427c2f4ac12ab34 (127.8.158.65:43095): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a7b3493272f64b74a5e2e2bea5319d7b: Network error: Client connection negotiation failed: client connection to 127.8.158.65:43095: connect: Connection refused (error 111)
I20260929 19:03:24.106686 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 16027
I20260929 19:03:24.112071 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 15379
I20260929 19:03:24.116827 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 15578
2026-09-29T19:03:24Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5689 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:03:24 dist-test-slave-nmsz krb5kdc[16283](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:03:24 dist-test-slave-nmsz krb5kdc[16283](info): set up 2 sockets
Sep 29 19:03:24 dist-test-slave-nmsz krb5kdc[16283](info): commencing operation
krb5kdc: starting...
W20260929 19:03:25.155237 8825 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Sep 29 19:03:25 dist-test-slave-nmsz krb5kdc[16283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:03:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:03:25Z Disabled control of system clock
WARNING: no policy specified for oryx/127.8.158.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.158.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.158.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.126.keytab.
Entry for principal oryx/127.8.158.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.126.keytab.
I20260929 19:03:25.273224 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:43799
--webserver_interface=127.8.158.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.126.keytab
--principal=oryx/127.8.158.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:45169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:43799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:25.352542 16296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:25.352744 16296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:25.352768 16296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:25.353839 16296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:03:25.353873 16296 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 19:03:25.353885 16296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:25.353899 16296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:25.353910 16296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:03:25.353922 16296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:03:25.355430 16296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:45169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:43799
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.126.keytab
--principal=oryx/127.8.158.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:43799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.158.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:25.355615 16296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:25.355769 16296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:25.360479 16304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:25.360486 16302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:25.360795 16296 server_base.cc:1034] running on GCE node
W20260929 19:03:25.360834 16301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:25.361090 16296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:25.361316 16296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:25.362479 16296 hybrid_clock.cc:648] HybridClock initialized: now 1790708605362454 us; error 30 us; skew 500 ppm
Sep 29 19:03:25 dist-test-slave-nmsz krb5kdc[16283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708605, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.158.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:25.368203 16296 init.cc:377] Logged in from keytab as oryx/127.8.158.126@KRBTEST.COM (short username oryx)
I20260929 19:03:25.368602 16296 webserver.cc:460] Webserver started at http://127.8.158.126:43209/ using document root <none> and password file <none>
I20260929 19:03:25.368830 16296 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:25.368896 16296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:25.369042 16296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:25.370074 16296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "61f4a227b7324cb584378135dfc9874a"
format_stamp: "Formatted at 2026-09-29 19:03:25 on dist-test-slave-nmsz"
server_key: "4ecd4050dc01ea60202b57a4548580e5"
server_key_iv: "bbd4caebc8a56f1a73cdaa0d372e3587"
server_key_version: "encryptionkey@0"
I20260929 19:03:25.370456 16296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "61f4a227b7324cb584378135dfc9874a"
format_stamp: "Formatted at 2026-09-29 19:03:25 on dist-test-slave-nmsz"
server_key: "4ecd4050dc01ea60202b57a4548580e5"
server_key_iv: "bbd4caebc8a56f1a73cdaa0d372e3587"
server_key_version: "encryptionkey@0"
I20260929 19:03:25.371752 16296 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:03:25.372445 16311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:25.372783 16296 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:03:25.372882 16296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "61f4a227b7324cb584378135dfc9874a"
format_stamp: "Formatted at 2026-09-29 19:03:25 on dist-test-slave-nmsz"
server_key: "4ecd4050dc01ea60202b57a4548580e5"
server_key_iv: "bbd4caebc8a56f1a73cdaa0d372e3587"
server_key_version: "encryptionkey@0"
I20260929 19:03:25.372956 16296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:25.385883 16296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:25.388334 16296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:25.388501 16296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:25.393250 16296 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.126:43799
I20260929 19:03:25.393302 16373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.126:43799 every 8 connection(s)
I20260929 19:03:25.393608 16296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:03:25.394212 16374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:25.396302 16374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61f4a227b7324cb584378135dfc9874a: Bootstrap starting.
I20260929 19:03:25.396875 16374 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61f4a227b7324cb584378135dfc9874a: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:25.397164 16374 log.cc:824] T 00000000000000000000000000000000 P 61f4a227b7324cb584378135dfc9874a: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:25.397552 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 16296
I20260929 19:03:25.397675 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:03:25.397893 8825 external_mini_cluster.cc:1428] Setting key 64e76a7af62bc04a0a017d8e7eafaacf
I20260929 19:03:25.397977 16374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61f4a227b7324cb584378135dfc9874a: No bootstrap required, opened a new log
I20260929 19:03:25.400130 16374 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61f4a227b7324cb584378135dfc9874a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f4a227b7324cb584378135dfc9874a" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 43799 } }
I20260929 19:03:25.400282 16374 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61f4a227b7324cb584378135dfc9874a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:25.400326 16374 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61f4a227b7324cb584378135dfc9874a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61f4a227b7324cb584378135dfc9874a, State: Initialized, Role: FOLLOWER
I20260929 19:03:25.400439 16374 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61f4a227b7324cb584378135dfc9874a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f4a227b7324cb584378135dfc9874a" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 43799 } }
I20260929 19:03:25.400533 16374 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61f4a227b7324cb584378135dfc9874a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:03:25.400578 16374 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61f4a227b7324cb584378135dfc9874a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:03:25.400655 16374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61f4a227b7324cb584378135dfc9874a [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:25.401156 16374 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61f4a227b7324cb584378135dfc9874a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f4a227b7324cb584378135dfc9874a" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 43799 } }
I20260929 19:03:25.401271 16374 leader_election.cc:302] T 00000000000000000000000000000000 P 61f4a227b7324cb584378135dfc9874a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61f4a227b7324cb584378135dfc9874a; no voters:
I20260929 19:03:25.401453 16374 leader_election.cc:288] T 00000000000000000000000000000000 P 61f4a227b7324cb584378135dfc9874a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:03:25.401481 16379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61f4a227b7324cb584378135dfc9874a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:03:25.401594 16379 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61f4a227b7324cb584378135dfc9874a [term 1 LEADER]: Becoming Leader. State: Replica: 61f4a227b7324cb584378135dfc9874a, State: Running, Role: LEADER
I20260929 19:03:25.401674 16379 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61f4a227b7324cb584378135dfc9874a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f4a227b7324cb584378135dfc9874a" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 43799 } }
I20260929 19:03:25.401711 16374 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61f4a227b7324cb584378135dfc9874a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:03:25.401988 16380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61f4a227b7324cb584378135dfc9874a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61f4a227b7324cb584378135dfc9874a. Latest consensus state: current_term: 1 leader_uuid: "61f4a227b7324cb584378135dfc9874a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f4a227b7324cb584378135dfc9874a" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 43799 } } }
I20260929 19:03:25.402086 16380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61f4a227b7324cb584378135dfc9874a [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:25.402349 16383 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:03:25.402361 16381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61f4a227b7324cb584378135dfc9874a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61f4a227b7324cb584378135dfc9874a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f4a227b7324cb584378135dfc9874a" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 43799 } } }
I20260929 19:03:25.402549 16381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61f4a227b7324cb584378135dfc9874a [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:25.402840 16383 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:03:25.404390 16383 catalog_manager.cc:1347] Generated new cluster ID: 9a632c56644c4c10a52e9d6921d27825
I20260929 19:03:25.404444 16383 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 19:03:25 dist-test-slave-nmsz krb5kdc[16283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM
I20260929 19:03:25.410827 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:25.398411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57150 (local address 127.8.158.126:43799)
0929 19:03:25.398611 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:25.398621 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:25.398663 (+ 42us) server_negotiation.cc:407] Connection header received
0929 19:03:25.400139 (+ 1476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:25.400146 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:25.400199 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:25.400366 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:25.401071 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:25.402376 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:25.403572 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:25.404156 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:25.404261 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:25.407142 (+ 2881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:25.407146 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:25.407146 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:25.407157 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:25.408958 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:25.409374 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:25.409377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:25.409378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:25.409491 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:25.409739 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:25.409740 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:25.409740 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:25.410019 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:25.410219 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:25.410382 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 19:03:25.410611 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":70,"threads_started":1}
I20260929 19:03:25.425377 16383 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:03:25.425846 16383 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:03:25.429454 16383 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61f4a227b7324cb584378135dfc9874a: Generated new TSK 0
I20260929 19:03:25.429621 16383 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.8.158.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.158.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.158.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.65.keytab.
Entry for principal oryx/127.8.158.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.65.keytab.
I20260929 19:03:25.460637 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.65:0
--local_ip_for_outbound_sockets=127.8.158.65
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:43799
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.65.keytab
--principal=oryx/127.8.158.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:45169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:25.538439 16400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:25.538615 16400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:25.538640 16400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:25.539770 16400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:25.539810 16400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:25.539830 16400 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 19:03:25.539884 16400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.65
I20260929 19:03:25.541317 16400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:45169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.65.keytab
--principal=oryx/127.8.158.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.158.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.158.126:43799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.65
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:25.541533 16400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:25.541774 16400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:25.546584 16406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:25.546566 16405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:25.546813 16400 server_base.cc:1034] running on GCE node
W20260929 19:03:25.546562 16408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:25.547138 16400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:25.547384 16400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:25.548560 16400 hybrid_clock.cc:648] HybridClock initialized: now 1790708605548555 us; error 51 us; skew 500 ppm
Sep 29 19:03:25 dist-test-slave-nmsz krb5kdc[16283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708605, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.158.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:25.551141 16400 init.cc:377] Logged in from keytab as oryx/127.8.158.65@KRBTEST.COM (short username oryx)
I20260929 19:03:25.551556 16400 webserver.cc:460] Webserver started at http://127.8.158.65:39429/ using document root <none> and password file <none>
I20260929 19:03:25.551788 16400 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:25.551864 16400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:25.552017 16400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:25.553054 16400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-0/data/instance:
uuid: "2c266e5fa9d94019907157fb874402b1"
format_stamp: "Formatted at 2026-09-29 19:03:25 on dist-test-slave-nmsz"
server_key: "aa97089b5be4aa3fd87aeea76d9a621f"
server_key_iv: "bf587cbb4daaa4f264e0f2d880ce23c4"
server_key_version: "encryptionkey@0"
I20260929 19:03:25.553431 16400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance:
uuid: "2c266e5fa9d94019907157fb874402b1"
format_stamp: "Formatted at 2026-09-29 19:03:25 on dist-test-slave-nmsz"
server_key: "aa97089b5be4aa3fd87aeea76d9a621f"
server_key_iv: "bf587cbb4daaa4f264e0f2d880ce23c4"
server_key_version: "encryptionkey@0"
I20260929 19:03:25.554818 16400 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 19:03:25.555480 16415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:25.555627 16400 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:25.555727 16400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-0/wal
uuid: "2c266e5fa9d94019907157fb874402b1"
format_stamp: "Formatted at 2026-09-29 19:03:25 on dist-test-slave-nmsz"
server_key: "aa97089b5be4aa3fd87aeea76d9a621f"
server_key_iv: "bf587cbb4daaa4f264e0f2d880ce23c4"
server_key_version: "encryptionkey@0"
I20260929 19:03:25.555802 16400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:25.566066 16400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:25.568239 16400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:25.568398 16400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:25.568873 16400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:25.568941 16400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:25.568982 16400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:25.569010 16400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:25.575667 16400 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.65:34403
I20260929 19:03:25.575857 16530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.65:34403 every 8 connection(s)
Sep 29 19:03:25 dist-test-slave-nmsz krb5kdc[16283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708605, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.158.65@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM
I20260929 19:03:25.576036 16400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
I20260929 19:03:25.578718 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:25.569324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:35289 (local address 127.8.158.126:43799)
0929 19:03:25.569371 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:25.569373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:25.569567 (+ 194us) server_negotiation.cc:407] Connection header received
0929 19:03:25.570048 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:25.570049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:25.570066 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:25.570117 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:25.570923 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:25.571750 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:25.573307 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:25.573659 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:25.573731 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:25.576453 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:25.576454 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:25.576454 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:25.576462 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:25.577600 (+ 1138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:25.577954 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:25.577955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:25.577955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:25.578003 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:25.578170 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:25.578170 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:25.578170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:25.578257 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:25.578362 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:25.578598 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 19:03:25.578659 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 19:03:25.578851 16424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:25.569321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:43799 (local address 127.8.158.65:35289)
0929 19:03:25.569449 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:25.569479 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:25.569942 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:25.570181 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:25.570184 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:25.570539 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:25.570810 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:25.570817 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:25.571865 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:25.571866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:25.573218 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:25.573220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:25.573746 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:25.573748 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:25.573914 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:25.574552 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:25.574562 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:25.576344 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:25.577669 (+ 1325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:25.577674 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:25.577676 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:25.577886 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:25.578045 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:25.578045 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:25.578045 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:25.578122 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:25.578366 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:25.578370 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:25.578488 (+ 118us) client_negotiation.cc:769] Sending connection context
0929 19:03:25.578571 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 19:03:25.578678 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":75,"threads_started":1}
I20260929 19:03:25.579295 16531 heartbeater.cc:347] Connected to a master server at 127.8.158.126:43799
I20260929 19:03:25.579382 16531 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:25.579550 16531 heartbeater.cc:511] Master 127.8.158.126:43799 requested a full tablet report, sending...
I20260929 19:03:25.580022 16327 ts_manager.cc:194] Registered new tserver with Master: 2c266e5fa9d94019907157fb874402b1 (127.8.158.65:34403)
I20260929 19:03:25.580940 16327 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.158.65@KRBTEST.COM'} at 127.8.158.65:35289
I20260929 19:03:25.584916 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 16400
I20260929 19:03:25.585026 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance
I20260929 19:03:25.585232 8825 external_mini_cluster.cc:1428] Setting key 80bd22b171ce8015f250c48d47b04835
I20260929 19:03:25.585341 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:25.580276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:54463 (local address 127.8.158.126:43799)
0929 19:03:25.580315 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:25.580317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:25.580402 (+ 85us) server_negotiation.cc:407] Connection header received
0929 19:03:25.580463 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:25.580464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:25.580471 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:25.580509 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:25.580807 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:25.581402 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:25.582144 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:25.582400 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:25.582461 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:25.583272 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:25.583273 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:25.583274 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:25.583278 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:25.584360 (+ 1082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:25.584637 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:25.584638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:25.584638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:25.584682 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:25.584831 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:25.584832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:25.584832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:25.584957 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:25.585037 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:25.585184 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 19:03:25.585253 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 19:03:25.585340 16537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:25.580194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:43799 (local address 127.8.158.65:54463)
0929 19:03:25.580338 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:25.580350 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:25.580410 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:25.580520 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:25.580521 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:25.580696 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:25.580750 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:25.580751 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:25.581489 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:25.581490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:25.582067 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:25.582069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:25.582462 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:25.582462 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:25.582508 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:25.582935 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:25.582940 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:25.583193 (+ 253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:25.584426 (+ 1233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:25.584426 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:25.584426 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:25.584576 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:25.584722 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:25.584722 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:25.584722 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:25.584761 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:25.585039 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:25.585040 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:25.585086 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 19:03:25.585156 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 19:03:25.585248 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
WARNING: no policy specified for oryx/127.8.158.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.158.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.158.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.66.keytab.
Entry for principal oryx/127.8.158.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.66.keytab.
I20260929 19:03:25.608806 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.66:0
--local_ip_for_outbound_sockets=127.8.158.66
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:43799
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.66.keytab
--principal=oryx/127.8.158.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:45169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:25.686483 16540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:25.686691 16540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:25.686726 16540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:25.687834 16540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:25.687897 16540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:25.687933 16540 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 19:03:25.687984 16540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.66
I20260929 19:03:25.689411 16540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:45169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.66.keytab
--principal=oryx/127.8.158.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.158.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.158.126:43799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.66
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:25.689675 16540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:25.689889 16540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:25.694615 16545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:25.694644 16548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:25.694808 16540 server_base.cc:1034] running on GCE node
W20260929 19:03:25.694626 16546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:25.695079 16540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:25.695310 16540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:25.696460 16540 hybrid_clock.cc:648] HybridClock initialized: now 1790708605696447 us; error 29 us; skew 500 ppm
Sep 29 19:03:25 dist-test-slave-nmsz krb5kdc[16283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708605, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.158.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:25.699014 16540 init.cc:377] Logged in from keytab as oryx/127.8.158.66@KRBTEST.COM (short username oryx)
I20260929 19:03:25.699446 16540 webserver.cc:460] Webserver started at http://127.8.158.66:33669/ using document root <none> and password file <none>
I20260929 19:03:25.699673 16540 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:25.699753 16540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:25.699905 16540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:25.700989 16540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-1/data/instance:
uuid: "720cc0480d0e4badabac18d7ff7e8bf4"
format_stamp: "Formatted at 2026-09-29 19:03:25 on dist-test-slave-nmsz"
server_key: "9b45afd8dff0862eff0070d9b73dc912"
server_key_iv: "13607bc368f3ff2852c5a535deb2cc1f"
server_key_version: "encryptionkey@0"
I20260929 19:03:25.701437 16540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance:
uuid: "720cc0480d0e4badabac18d7ff7e8bf4"
format_stamp: "Formatted at 2026-09-29 19:03:25 on dist-test-slave-nmsz"
server_key: "9b45afd8dff0862eff0070d9b73dc912"
server_key_iv: "13607bc368f3ff2852c5a535deb2cc1f"
server_key_version: "encryptionkey@0"
I20260929 19:03:25.703016 16540 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:03:25.703799 16555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:25.703992 16540 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:25.704085 16540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-1/wal
uuid: "720cc0480d0e4badabac18d7ff7e8bf4"
format_stamp: "Formatted at 2026-09-29 19:03:25 on dist-test-slave-nmsz"
server_key: "9b45afd8dff0862eff0070d9b73dc912"
server_key_iv: "13607bc368f3ff2852c5a535deb2cc1f"
server_key_version: "encryptionkey@0"
I20260929 19:03:25.704160 16540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:25.719306 16540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:25.721225 16540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:25.721390 16540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:25.721911 16540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:25.721951 16540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:25.722007 16540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:25.722038 16540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 19:03:25 dist-test-slave-nmsz krb5kdc[16283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708605, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.158.66@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM
I20260929 19:03:25.729377 16540 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.66:36373
I20260929 19:03:25.729446 16670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.66:36373 every 8 connection(s)
I20260929 19:03:25.729813 16540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
I20260929 19:03:25.732112 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:25.722568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:56783 (local address 127.8.158.126:43799)
0929 19:03:25.722609 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:25.722611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:25.722874 (+ 263us) server_negotiation.cc:407] Connection header received
0929 19:03:25.723399 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:25.723400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:25.723414 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:25.723464 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:25.724076 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:25.724883 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:25.726406 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:25.726614 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:25.726697 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:25.729251 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:25.729253 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:25.729253 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:25.729261 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:25.730968 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:25.731317 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:25.731318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:25.731318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:25.731369 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:25.731599 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:25.731600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:25.731600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:25.731712 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:25.731779 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:25.731970 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 19:03:25.732032 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 19:03:25.732227 16564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:25.722580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:43799 (local address 127.8.158.66:56783)
0929 19:03:25.722763 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:25.722794 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:25.723289 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:25.723525 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:25.723528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:25.723819 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:25.723986 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:25.723990 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:25.724970 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:25.724971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:25.726337 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:25.726340 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:25.726719 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:25.726720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:25.726841 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:25.727324 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:25.727334 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:25.729120 (+ 1786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:25.731041 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:25.731044 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:25.731046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:25.731247 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:25.731426 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:25.731427 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:25.731427 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:25.731512 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:25.731779 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:25.731782 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:25.731879 (+ 97us) client_negotiation.cc:769] Sending connection context
0929 19:03:25.731957 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 19:03:25.732060 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":91,"threads_started":1}
I20260929 19:03:25.732681 16671 heartbeater.cc:347] Connected to a master server at 127.8.158.126:43799
I20260929 19:03:25.732785 16671 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:25.732942 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 16540
I20260929 19:03:25.732959 16671 heartbeater.cc:511] Master 127.8.158.126:43799 requested a full tablet report, sending...
I20260929 19:03:25.733068 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance
I20260929 19:03:25.733286 8825 external_mini_cluster.cc:1428] Setting key b16f85f2f5daac04d52a5af39d17e338
I20260929 19:03:25.733331 16327 ts_manager.cc:194] Registered new tserver with Master: 720cc0480d0e4badabac18d7ff7e8bf4 (127.8.158.66:36373)
I20260929 19:03:25.733798 16327 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.158.66@KRBTEST.COM'} at 127.8.158.66:56783
I20260929 19:03:25.739603 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:25.734116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:32849 (local address 127.8.158.126:43799)
0929 19:03:25.734152 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:25.734154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:25.734348 (+ 194us) server_negotiation.cc:407] Connection header received
0929 19:03:25.734427 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:25.734427 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:25.734437 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:25.734491 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:25.734792 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:25.735228 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:25.735905 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:25.736103 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:25.736183 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:25.737042 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:25.737044 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:25.737044 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:25.737050 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:25.738473 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:25.738818 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:25.738819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:25.738819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:25.738871 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:25.739074 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:25.739075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:25.739076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:25.739216 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:25.739294 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:25.739448 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 19:03:25.739515 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260929 19:03:25.739609 16677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:25.734089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:43799 (local address 127.8.158.66:32849)
0929 19:03:25.734257 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:25.734266 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:25.734379 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:25.734497 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:25.734497 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:25.734662 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:25.734709 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:25.734710 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:25.735315 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:25.735316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:25.735812 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:25.735813 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:25.736185 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:25.736185 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:25.736257 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:25.736675 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:25.736680 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:25.736939 (+ 259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:25.738563 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:25.738564 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:25.738564 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:25.738733 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:25.738950 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:25.738951 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:25.738951 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:25.739012 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:25.739285 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:25.739286 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:25.739328 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 19:03:25.739422 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 19:03:25.739517 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":109,"threads_started":1}
WARNING: no policy specified for oryx/127.8.158.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.158.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.158.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.67.keytab.
Entry for principal oryx/127.8.158.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.67.keytab.
I20260929 19:03:25.757550 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.67:0
--local_ip_for_outbound_sockets=127.8.158.67
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:43799
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.67.keytab
--principal=oryx/127.8.158.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:45169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:25.835928 16680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:25.836109 16680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:25.836131 16680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:25.837170 16680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:25.837208 16680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:25.837229 16680 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 19:03:25.837283 16680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.67
I20260929 19:03:25.838775 16680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:45169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.67.keytab
--principal=oryx/127.8.158.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.158.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.158.126:43799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.67
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:25.839003 16680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:25.839244 16680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:25.844201 16686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:25.844218 16688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:25.844218 16685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:25.844394 16680 server_base.cc:1034] running on GCE node
I20260929 19:03:25.844609 16680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:25.844846 16680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:25.846009 16680 hybrid_clock.cc:648] HybridClock initialized: now 1790708605846004 us; error 42 us; skew 500 ppm
Sep 29 19:03:25 dist-test-slave-nmsz krb5kdc[16283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708605, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.158.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:25.848394 16680 init.cc:377] Logged in from keytab as oryx/127.8.158.67@KRBTEST.COM (short username oryx)
I20260929 19:03:25.848839 16680 webserver.cc:460] Webserver started at http://127.8.158.67:46811/ using document root <none> and password file <none>
I20260929 19:03:25.849047 16680 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:25.849113 16680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:25.849254 16680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:25.850211 16680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-2/data/instance:
uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159"
format_stamp: "Formatted at 2026-09-29 19:03:25 on dist-test-slave-nmsz"
server_key: "79b149e397f9df2a4403c1407b3e4436"
server_key_iv: "6ebe8af1a820fb4f93f6b734897d54db"
server_key_version: "encryptionkey@0"
I20260929 19:03:25.850639 16680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance:
uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159"
format_stamp: "Formatted at 2026-09-29 19:03:25 on dist-test-slave-nmsz"
server_key: "79b149e397f9df2a4403c1407b3e4436"
server_key_iv: "6ebe8af1a820fb4f93f6b734897d54db"
server_key_version: "encryptionkey@0"
I20260929 19:03:25.851975 16680 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:25.852722 16695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:25.852908 16680 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:25.853003 16680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-2/wal
uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159"
format_stamp: "Formatted at 2026-09-29 19:03:25 on dist-test-slave-nmsz"
server_key: "79b149e397f9df2a4403c1407b3e4436"
server_key_iv: "6ebe8af1a820fb4f93f6b734897d54db"
server_key_version: "encryptionkey@0"
I20260929 19:03:25.853084 16680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:25.873670 16680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:25.875751 16680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:25.875905 16680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:25.876451 16680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:25.876508 16680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:25.876549 16680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:25.876577 16680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 19:03:25 dist-test-slave-nmsz krb5kdc[16283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708605, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.158.67@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM
I20260929 19:03:25.883716 16680 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.67:34541
I20260929 19:03:25.884043 16680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
I20260929 19:03:25.884532 16810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.67:34541 every 8 connection(s)
I20260929 19:03:25.886655 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:25.876887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:45661 (local address 127.8.158.126:43799)
0929 19:03:25.876956 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:25.876959 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:25.877140 (+ 181us) server_negotiation.cc:407] Connection header received
0929 19:03:25.877642 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:25.877643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:25.877654 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:25.877715 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:25.878379 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:25.878940 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:25.880161 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:25.880429 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:25.880495 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:25.883720 (+ 3225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:25.883721 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:25.883722 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:25.883728 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:25.885201 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:25.885621 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:25.885622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:25.885623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:25.885687 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:25.885970 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:25.885971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:25.885971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:25.886083 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:25.886157 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:25.886469 (+ 312us) server_negotiation.cc:299] Negotiation successful
0929 19:03:25.886548 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 19:03:25.886826 16704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:25.876873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:43799 (local address 127.8.158.67:45661)
0929 19:03:25.877031 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:25.877068 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:25.877534 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:25.877794 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:25.877797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:25.878084 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:25.878253 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:25.878257 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:25.879041 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:25.879042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:25.880065 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:25.880068 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:25.880509 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:25.880511 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:25.880684 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:25.881221 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:25.881238 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:25.883617 (+ 2379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:25.885305 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:25.885312 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:25.885313 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:25.885531 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:25.885773 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:25.885774 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:25.885774 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:25.885887 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:25.886150 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:25.886155 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:25.886295 (+ 140us) client_negotiation.cc:769] Sending connection context
0929 19:03:25.886447 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 19:03:25.886606 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":91,"threads_started":1}
I20260929 19:03:25.887290 16811 heartbeater.cc:347] Connected to a master server at 127.8.158.126:43799
I20260929 19:03:25.887383 16811 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:25.887547 16811 heartbeater.cc:511] Master 127.8.158.126:43799 requested a full tablet report, sending...
I20260929 19:03:25.887910 16328 ts_manager.cc:194] Registered new tserver with Master: 9ea6b6f3d04e4b16aa63c8ad5dcee159 (127.8.158.67:34541)
I20260929 19:03:25.888289 16328 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.158.67@KRBTEST.COM'} at 127.8.158.67:45661
I20260929 19:03:25.892133 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 16680
I20260929 19:03:25.892242 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance
I20260929 19:03:25.892454 8825 external_mini_cluster.cc:1428] Setting key 539b63c9bdd3f5006e29eb6a51146e1c
I20260929 19:03:25.893215 8825 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:03:25.894189 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:25.888343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:57377 (local address 127.8.158.126:43799)
0929 19:03:25.888387 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:25.888389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:25.888447 (+ 58us) server_negotiation.cc:407] Connection header received
0929 19:03:25.888522 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:25.888523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:25.888533 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:25.888592 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:25.888909 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:25.889454 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:25.890162 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:25.890447 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:25.890504 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:25.891387 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:25.891389 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:25.891389 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:25.891396 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:25.893112 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:25.893455 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:25.893456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:25.893457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:25.893507 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:25.893711 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:25.893712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:25.893712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:25.893819 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:25.893889 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:25.894036 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 19:03:25.894104 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 19:03:25.894204 16817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:25.888267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:43799 (local address 127.8.158.67:57377)
0929 19:03:25.888379 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:25.888389 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:25.888472 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:25.888593 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:25.888594 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:25.888775 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:25.888826 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:25.888828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:25.889559 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:25.889560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:25.890076 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:25.890077 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:25.890503 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:25.890504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:25.890565 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:25.890993 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:25.890997 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:25.891277 (+ 280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:25.893181 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:25.893182 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:25.893182 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:25.893386 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:25.893586 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:25.893586 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:25.893587 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:25.893648 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:25.893882 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:25.893883 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:25.893926 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 19:03:25.894012 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 19:03:25.894106 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":62,"threads_started":1}
Sep 29 19:03:25 dist-test-slave-nmsz krb5kdc[16283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708605, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 19:03:25 dist-test-slave-nmsz krb5kdc[16283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:25 dist-test-slave-nmsz krb5kdc[16283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 19:03:25.914542 16378 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:25.908822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57164 (local address 127.8.158.126:43799)
0929 19:03:25.908881 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:25.908883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:25.908892 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:03:25.908918 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:25.908919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:25.908930 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:25.909026 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:25.909651 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:25.910163 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:25.911074 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:25.911274 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:25.911352 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:25.914442 (+ 3090us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57164: BlockingRecv error: recv got EOF from 127.0.0.1:57164 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 29 19:03:25 dist-test-slave-nmsz krb5kdc[16283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708605, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM
I20260929 19:03:25.921535 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:25.915318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57174 (local address 127.8.158.126:43799)
0929 19:03:25.915362 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:25.915365 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:25.915396 (+ 31us) server_negotiation.cc:407] Connection header received
0929 19:03:25.915454 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:25.915454 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:25.915461 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:25.915514 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:25.915968 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:25.916423 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:25.917071 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:25.917236 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:25.917283 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:25.919295 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:25.919297 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:25.919297 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:25.919304 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:25.920463 (+ 1159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:25.920812 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:25.920813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:25.920813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:25.920871 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:25.921066 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:25.921067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:25.921067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:25.921169 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:25.921265 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:25.921369 (+ 104us) server_negotiation.cc:299] Negotiation successful
0929 19:03:25.921440 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 19:03:25.923081 16328 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57174:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:03:25.924738 16328 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:03:25.931224 16830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:25.926827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.67:34541 (local address 127.0.0.1:34246)
0929 19:03:25.927416 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:25.927427 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:25.927492 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:25.927740 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:25.927742 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:25.927744 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:25.927812 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:25.927814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:25.929143 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:25.929144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:25.930242 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:25.930244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:25.930936 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:25.930937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:25.931017 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:25.931030 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 19:03:25.931104 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 19:03:25.931152 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"mutex_wait_us":2,"thread_start_us":66,"threads_started":1}
I20260929 19:03:25.931265 16835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:25.927219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34246 (local address 127.8.158.67:34541)
0929 19:03:25.927390 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:25.927395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:25.927476 (+ 81us) server_negotiation.cc:407] Connection header received
0929 19:03:25.927572 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:25.927575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:25.927594 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:25.927673 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:25.927905 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:25.929062 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:25.930347 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:25.930834 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:25.930933 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:25.931009 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:25.931117 (+ 108us) server_negotiation.cc:299] Negotiation successful
0929 19:03:25.931163 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":110,"threads_started":1}
I20260929 19:03:25.931524 16831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:25.927034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:34403 (local address 127.0.0.1:41502)
0929 19:03:25.927580 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:25.927590 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:25.927697 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:25.927998 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:25.927998 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:25.928000 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:25.928033 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:25.928034 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:25.928845 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:25.928846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:25.930105 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:25.930107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:25.931249 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:25.931250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:25.931321 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:25.931327 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 19:03:25.931392 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 19:03:25.931445 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":50,"threads_started":1}
I20260929 19:03:25.931898 16834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:25.927145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41502 (local address 127.8.158.65:34403)
0929 19:03:25.927300 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:25.927305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:25.927860 (+ 555us) server_negotiation.cc:407] Connection header received
0929 19:03:25.927908 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:25.927910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:25.927927 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:25.928000 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:25.928087 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:25.928750 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:25.930210 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:25.931051 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:25.931130 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:25.931209 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:25.931713 (+ 504us) server_negotiation.cc:299] Negotiation successful
0929 19:03:25.931761 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":97,"threads_started":1}
I20260929 19:03:25.932094 16832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:25.926827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.66:36373 (local address 127.0.0.1:49494)
0929 19:03:25.927331 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:25.927366 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:25.927463 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:25.927642 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:25.927645 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:25.927646 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:25.927704 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:25.927706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:25.929161 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:25.929162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:25.930264 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:25.930266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:25.931495 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:25.931496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:25.931756 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:25.931762 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 19:03:25.931826 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 19:03:25.931869 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":66,"threads_started":1}
I20260929 19:03:25.932003 16734 tablet_service.cc:1467] Processing CreateTablet for tablet f6f81f9570fd4f078644ce7c5022f908 (DEFAULT_TABLE table=test-table [id=cdfd5e2042964b4ea9e4bbef2549c518]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:25.932346 16734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6f81f9570fd4f078644ce7c5022f908. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:25.932498 16465 tablet_service.cc:1467] Processing CreateTablet for tablet f6f81f9570fd4f078644ce7c5022f908 (DEFAULT_TABLE table=test-table [id=cdfd5e2042964b4ea9e4bbef2549c518]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:25.932842 16465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6f81f9570fd4f078644ce7c5022f908. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:25.934438 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:25.927033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49494 (local address 127.8.158.66:36373)
0929 19:03:25.927189 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:25.927193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:25.927445 (+ 252us) server_negotiation.cc:407] Connection header received
0929 19:03:25.927542 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:25.927545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:25.927560 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:25.927632 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:25.927831 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:25.929050 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:25.930405 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:25.931147 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:25.931190 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:25.931263 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:25.931851 (+ 588us) server_negotiation.cc:299] Negotiation successful
0929 19:03:25.931889 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":85,"threads_started":1}
I20260929 19:03:25.935063 16605 tablet_service.cc:1467] Processing CreateTablet for tablet f6f81f9570fd4f078644ce7c5022f908 (DEFAULT_TABLE table=test-table [id=cdfd5e2042964b4ea9e4bbef2549c518]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:25.935367 16605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6f81f9570fd4f078644ce7c5022f908. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:25.936038 16836 tablet_bootstrap.cc:492] T f6f81f9570fd4f078644ce7c5022f908 P 9ea6b6f3d04e4b16aa63c8ad5dcee159: Bootstrap starting.
I20260929 19:03:25.936479 16837 tablet_bootstrap.cc:492] T f6f81f9570fd4f078644ce7c5022f908 P 2c266e5fa9d94019907157fb874402b1: Bootstrap starting.
I20260929 19:03:25.936852 16836 tablet_bootstrap.cc:654] T f6f81f9570fd4f078644ce7c5022f908 P 9ea6b6f3d04e4b16aa63c8ad5dcee159: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:25.937038 16837 tablet_bootstrap.cc:654] T f6f81f9570fd4f078644ce7c5022f908 P 2c266e5fa9d94019907157fb874402b1: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:25.937198 16836 log.cc:824] T f6f81f9570fd4f078644ce7c5022f908 P 9ea6b6f3d04e4b16aa63c8ad5dcee159: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:25.937386 16837 log.cc:824] T f6f81f9570fd4f078644ce7c5022f908 P 2c266e5fa9d94019907157fb874402b1: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:25.938042 16836 tablet_bootstrap.cc:492] T f6f81f9570fd4f078644ce7c5022f908 P 9ea6b6f3d04e4b16aa63c8ad5dcee159: No bootstrap required, opened a new log
I20260929 19:03:25.938130 16836 ts_tablet_manager.cc:1397] T f6f81f9570fd4f078644ce7c5022f908 P 9ea6b6f3d04e4b16aa63c8ad5dcee159: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:03:25.938270 16837 tablet_bootstrap.cc:492] T f6f81f9570fd4f078644ce7c5022f908 P 2c266e5fa9d94019907157fb874402b1: No bootstrap required, opened a new log
I20260929 19:03:25.938416 16837 ts_tablet_manager.cc:1397] T f6f81f9570fd4f078644ce7c5022f908 P 2c266e5fa9d94019907157fb874402b1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:03:25.939318 16840 tablet_bootstrap.cc:492] T f6f81f9570fd4f078644ce7c5022f908 P 720cc0480d0e4badabac18d7ff7e8bf4: Bootstrap starting.
I20260929 19:03:25.939852 16840 tablet_bootstrap.cc:654] T f6f81f9570fd4f078644ce7c5022f908 P 720cc0480d0e4badabac18d7ff7e8bf4: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:25.940156 16840 log.cc:824] T f6f81f9570fd4f078644ce7c5022f908 P 720cc0480d0e4badabac18d7ff7e8bf4: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:25.940107 16836 raft_consensus.cc:348] T f6f81f9570fd4f078644ce7c5022f908 P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } } peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } }
I20260929 19:03:25.940277 16836 raft_consensus.cc:374] T f6f81f9570fd4f078644ce7c5022f908 P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:25.940320 16836 raft_consensus.cc:729] T f6f81f9570fd4f078644ce7c5022f908 P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ea6b6f3d04e4b16aa63c8ad5dcee159, State: Initialized, Role: FOLLOWER
I20260929 19:03:25.940342 16837 raft_consensus.cc:348] T f6f81f9570fd4f078644ce7c5022f908 P 2c266e5fa9d94019907157fb874402b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } } peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } }
I20260929 19:03:25.940423 16836 consensus_queue.cc:260] T f6f81f9570fd4f078644ce7c5022f908 P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } } peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } }
I20260929 19:03:25.940474 16837 raft_consensus.cc:374] T f6f81f9570fd4f078644ce7c5022f908 P 2c266e5fa9d94019907157fb874402b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:25.940507 16837 raft_consensus.cc:729] T f6f81f9570fd4f078644ce7c5022f908 P 2c266e5fa9d94019907157fb874402b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c266e5fa9d94019907157fb874402b1, State: Initialized, Role: FOLLOWER
I20260929 19:03:25.940634 16837 consensus_queue.cc:260] T f6f81f9570fd4f078644ce7c5022f908 P 2c266e5fa9d94019907157fb874402b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } } peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } }
I20260929 19:03:25.940675 16836 ts_tablet_manager.cc:1428] T f6f81f9570fd4f078644ce7c5022f908 P 9ea6b6f3d04e4b16aa63c8ad5dcee159: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 19:03:25.940783 16811 heartbeater.cc:503] Master 127.8.158.126:43799 was elected leader, sending a full tablet report...
I20260929 19:03:25.940918 16840 tablet_bootstrap.cc:492] T f6f81f9570fd4f078644ce7c5022f908 P 720cc0480d0e4badabac18d7ff7e8bf4: No bootstrap required, opened a new log
I20260929 19:03:25.940999 16840 ts_tablet_manager.cc:1397] T f6f81f9570fd4f078644ce7c5022f908 P 720cc0480d0e4badabac18d7ff7e8bf4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:03:25.941143 16837 ts_tablet_manager.cc:1428] T f6f81f9570fd4f078644ce7c5022f908 P 2c266e5fa9d94019907157fb874402b1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 19:03:25.941285 16531 heartbeater.cc:503] Master 127.8.158.126:43799 was elected leader, sending a full tablet report...
I20260929 19:03:25.943032 16840 raft_consensus.cc:348] T f6f81f9570fd4f078644ce7c5022f908 P 720cc0480d0e4badabac18d7ff7e8bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } } peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } }
I20260929 19:03:25.943156 16840 raft_consensus.cc:374] T f6f81f9570fd4f078644ce7c5022f908 P 720cc0480d0e4badabac18d7ff7e8bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:25.943178 16840 raft_consensus.cc:729] T f6f81f9570fd4f078644ce7c5022f908 P 720cc0480d0e4badabac18d7ff7e8bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 720cc0480d0e4badabac18d7ff7e8bf4, State: Initialized, Role: FOLLOWER
I20260929 19:03:25.943327 16840 consensus_queue.cc:260] T f6f81f9570fd4f078644ce7c5022f908 P 720cc0480d0e4badabac18d7ff7e8bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } } peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } }
I20260929 19:03:25.943605 16840 ts_tablet_manager.cc:1428] T f6f81f9570fd4f078644ce7c5022f908 P 720cc0480d0e4badabac18d7ff7e8bf4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 19:03:25.943634 16671 heartbeater.cc:503] Master 127.8.158.126:43799 was elected leader, sending a full tablet report...
W20260929 19:03:25.980711 16672 tablet.cc:2369] T f6f81f9570fd4f078644ce7c5022f908 P 720cc0480d0e4badabac18d7ff7e8bf4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 19:03:26.076763 16532 tablet.cc:2369] T f6f81f9570fd4f078644ce7c5022f908 P 2c266e5fa9d94019907157fb874402b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:03:26.129217 16844 raft_consensus.cc:482] T f6f81f9570fd4f078644ce7c5022f908 P 720cc0480d0e4badabac18d7ff7e8bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:03:26.129336 16844 raft_consensus.cc:504] T f6f81f9570fd4f078644ce7c5022f908 P 720cc0480d0e4badabac18d7ff7e8bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } } peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } }
I20260929 19:03:26.129734 16844 leader_election.cc:288] T f6f81f9570fd4f078644ce7c5022f908 P 720cc0480d0e4badabac18d7ff7e8bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ea6b6f3d04e4b16aa63c8ad5dcee159 (127.8.158.67:34541), 2c266e5fa9d94019907157fb874402b1 (127.8.158.65:34403)
I20260929 19:03:26.132680 16835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:26.130006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:43643 (local address 127.8.158.67:34541)
0929 19:03:26.130085 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:26.130087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:26.130096 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:03:26.130153 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:26.130154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:26.130164 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:26.130224 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:26.130426 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.130957 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.131943 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.132321 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.132409 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:26.132417 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:26.132585 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 19:03:26.132618 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 19:03:26.132680 16564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:26.129922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.67:34541 (local address 127.8.158.66:43643)
0929 19:03:26.129993 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:26.130007 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:26.130067 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:26.130237 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:26.130237 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:26.130239 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:26.130343 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:26.130345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.131043 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.131044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:26.131841 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:26.131844 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.132405 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.132406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:26.132501 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:26.132503 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 19:03:26.132573 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 19:03:26.132611 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":37}
I20260929 19:03:26.132746 16834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:26.130131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:59049 (local address 127.8.158.65:34403)
0929 19:03:26.130172 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:26.130174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:26.130275 (+ 101us) server_negotiation.cc:407] Connection header received
0929 19:03:26.130308 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:26.130335 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:26.130345 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:26.130406 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:26.130553 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.131029 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.131987 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.132398 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.132475 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:26.132483 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:26.132638 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 19:03:26.132679 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 19:03:26.132784 16845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:26.130057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:34403 (local address 127.8.158.66:59049)
0929 19:03:26.130205 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:26.130215 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:26.130262 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:26.130421 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:26.130422 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:26.130424 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:26.130487 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:26.130489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.131118 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.131119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:26.131867 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:26.131869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.132485 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.132486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:26.132565 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:26.132568 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:03:26.132634 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 19:03:26.132672 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
I20260929 19:03:26.133101 16756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6f81f9570fd4f078644ce7c5022f908" candidate_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" is_pre_election: true
I20260929 19:03:26.133132 16485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6f81f9570fd4f078644ce7c5022f908" candidate_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c266e5fa9d94019907157fb874402b1" is_pre_election: true
I20260929 19:03:26.133256 16756 raft_consensus.cc:2454] T f6f81f9570fd4f078644ce7c5022f908 P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 720cc0480d0e4badabac18d7ff7e8bf4 in term 0.
I20260929 19:03:26.133304 16485 raft_consensus.cc:2454] T f6f81f9570fd4f078644ce7c5022f908 P 2c266e5fa9d94019907157fb874402b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 720cc0480d0e4badabac18d7ff7e8bf4 in term 0.
I20260929 19:03:26.133456 16558 leader_election.cc:302] T f6f81f9570fd4f078644ce7c5022f908 P 720cc0480d0e4badabac18d7ff7e8bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 720cc0480d0e4badabac18d7ff7e8bf4, 9ea6b6f3d04e4b16aa63c8ad5dcee159; no voters:
I20260929 19:03:26.133601 16844 raft_consensus.cc:2781] T f6f81f9570fd4f078644ce7c5022f908 P 720cc0480d0e4badabac18d7ff7e8bf4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:03:26.133670 16844 raft_consensus.cc:482] T f6f81f9570fd4f078644ce7c5022f908 P 720cc0480d0e4badabac18d7ff7e8bf4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:03:26.133688 16844 raft_consensus.cc:3037] T f6f81f9570fd4f078644ce7c5022f908 P 720cc0480d0e4badabac18d7ff7e8bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:26.134653 16844 raft_consensus.cc:504] T f6f81f9570fd4f078644ce7c5022f908 P 720cc0480d0e4badabac18d7ff7e8bf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } } peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } }
I20260929 19:03:26.134779 16844 leader_election.cc:288] T f6f81f9570fd4f078644ce7c5022f908 P 720cc0480d0e4badabac18d7ff7e8bf4 [CANDIDATE]: Term 1 election: Requested vote from peers 9ea6b6f3d04e4b16aa63c8ad5dcee159 (127.8.158.67:34541), 2c266e5fa9d94019907157fb874402b1 (127.8.158.65:34403)
W20260929 19:03:26.134900 16812 tablet.cc:2369] T f6f81f9570fd4f078644ce7c5022f908 P 9ea6b6f3d04e4b16aa63c8ad5dcee159: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:03:26.134920 16756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6f81f9570fd4f078644ce7c5022f908" candidate_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159"
I20260929 19:03:26.134956 16485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6f81f9570fd4f078644ce7c5022f908" candidate_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c266e5fa9d94019907157fb874402b1"
I20260929 19:03:26.134994 16756 raft_consensus.cc:3037] T f6f81f9570fd4f078644ce7c5022f908 P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:26.135028 16485 raft_consensus.cc:3037] T f6f81f9570fd4f078644ce7c5022f908 P 2c266e5fa9d94019907157fb874402b1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:26.135641 16485 raft_consensus.cc:2454] T f6f81f9570fd4f078644ce7c5022f908 P 2c266e5fa9d94019907157fb874402b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 720cc0480d0e4badabac18d7ff7e8bf4 in term 1.
I20260929 19:03:26.135736 16756 raft_consensus.cc:2454] T f6f81f9570fd4f078644ce7c5022f908 P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 720cc0480d0e4badabac18d7ff7e8bf4 in term 1.
I20260929 19:03:26.135797 16558 leader_election.cc:302] T f6f81f9570fd4f078644ce7c5022f908 P 720cc0480d0e4badabac18d7ff7e8bf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c266e5fa9d94019907157fb874402b1, 720cc0480d0e4badabac18d7ff7e8bf4; no voters:
I20260929 19:03:26.135905 16844 raft_consensus.cc:2781] T f6f81f9570fd4f078644ce7c5022f908 P 720cc0480d0e4badabac18d7ff7e8bf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:03:26.136034 16844 raft_consensus.cc:686] T f6f81f9570fd4f078644ce7c5022f908 P 720cc0480d0e4badabac18d7ff7e8bf4 [term 1 LEADER]: Becoming Leader. State: Replica: 720cc0480d0e4badabac18d7ff7e8bf4, State: Running, Role: LEADER
I20260929 19:03:26.136137 16844 consensus_queue.cc:237] T f6f81f9570fd4f078644ce7c5022f908 P 720cc0480d0e4badabac18d7ff7e8bf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } } peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } }
I20260929 19:03:26.136940 16327 catalog_manager.cc:5495] T f6f81f9570fd4f078644ce7c5022f908 P 720cc0480d0e4badabac18d7ff7e8bf4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 720cc0480d0e4badabac18d7ff7e8bf4 (127.8.158.66). New cstate: current_term: 1 leader_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } health_report { overall_health: UNKNOWN } } }
I20260929 19:03:26.160048 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:26.157599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49508 (local address 127.8.158.66:36373)
0929 19:03:26.157640 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:26.157642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:26.157659 (+ 17us) server_negotiation.cc:407] Connection header received
0929 19:03:26.157689 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:26.157690 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:26.157706 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:26.157791 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:03:26.157890 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.158475 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.159245 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.159476 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.159559 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:26.159720 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:03:26.159824 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:03:26.159915 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:26.159956 (+ 41us) server_negotiation.cc:299] Negotiation successful
0929 19:03:26.159974 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 19:03:26.161595 16485 raft_consensus.cc:1261] T f6f81f9570fd4f078644ce7c5022f908 P 2c266e5fa9d94019907157fb874402b1 [term 1 FOLLOWER]: Refusing update from remote peer 720cc0480d0e4badabac18d7ff7e8bf4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:03:26.161672 16756 raft_consensus.cc:1261] T f6f81f9570fd4f078644ce7c5022f908 P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [term 1 FOLLOWER]: Refusing update from remote peer 720cc0480d0e4badabac18d7ff7e8bf4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:03:26.161933 16844 consensus_queue.cc:1035] T f6f81f9570fd4f078644ce7c5022f908 P 720cc0480d0e4badabac18d7ff7e8bf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:03:26.162065 16846 consensus_queue.cc:1035] T f6f81f9570fd4f078644ce7c5022f908 P 720cc0480d0e4badabac18d7ff7e8bf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:03:26.164728 16850 mvcc.cc:205] Tried to move back new op lower bound from 7334742450835636224 to 7334742450734256128. Current Snapshot: MvccSnapshot[applied={T|T < 7334742450835636224}]
I20260929 19:03:26.164808 16849 mvcc.cc:205] Tried to move back new op lower bound from 7334742450835636224 to 7334742450734256128. Current Snapshot: MvccSnapshot[applied={T|T < 7334742450835636224 or (T in {7334742450835636224})}]
I20260929 19:03:26.165951 16851 mvcc.cc:205] Tried to move back new op lower bound from 7334742450835636224 to 7334742450734256128. Current Snapshot: MvccSnapshot[applied={T|T < 7334742450835636224}]
I20260929 19:03:26.166162 16327 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:03:26.166245 16327 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:03:26.167068 16327 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 61f4a227b7324cb584378135dfc9874a: Sending DeleteTablet for 3 replicas of tablet f6f81f9570fd4f078644ce7c5022f908
I20260929 19:03:26.167387 16734 tablet_service.cc:1514] Processing DeleteTablet for tablet f6f81f9570fd4f078644ce7c5022f908 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:26 UTC) from {username='oryx', principal='oryx/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:34246
I20260929 19:03:26.167435 16605 tablet_service.cc:1514] Processing DeleteTablet for tablet f6f81f9570fd4f078644ce7c5022f908 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:26 UTC) from {username='oryx', principal='oryx/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:49494
I20260929 19:03:26.167441 16465 tablet_service.cc:1514] Processing DeleteTablet for tablet f6f81f9570fd4f078644ce7c5022f908 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:26 UTC) from {username='oryx', principal='oryx/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:41502
I20260929 19:03:26.167743 16861 tablet_replica.cc:330] T f6f81f9570fd4f078644ce7c5022f908 P 720cc0480d0e4badabac18d7ff7e8bf4: stopping tablet replica
I20260929 19:03:26.167842 16861 raft_consensus.cc:2229] T f6f81f9570fd4f078644ce7c5022f908 P 720cc0480d0e4badabac18d7ff7e8bf4 [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:03:26.167814 16327 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57174:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:03:26.167928 16327 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:03:26.167950 16861 raft_consensus.cc:2258] T f6f81f9570fd4f078644ce7c5022f908 P 720cc0480d0e4badabac18d7ff7e8bf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:03:26.168129 16859 tablet_replica.cc:330] T f6f81f9570fd4f078644ce7c5022f908 P 9ea6b6f3d04e4b16aa63c8ad5dcee159: stopping tablet replica
I20260929 19:03:26.168272 16859 raft_consensus.cc:2229] T f6f81f9570fd4f078644ce7c5022f908 P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:03:26.168403 16861 ts_tablet_manager.cc:1905] T f6f81f9570fd4f078644ce7c5022f908 P 720cc0480d0e4badabac18d7ff7e8bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:03:26.168942 16859 raft_consensus.cc:2258] T f6f81f9570fd4f078644ce7c5022f908 P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:03:26.169150 16465 tablet_service.cc:1467] Processing CreateTablet for tablet ce6409f165134929b198d5b645f9a119 (DEFAULT_TABLE table=test-table [id=41cfe450a54040239425f359f49d3350]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:26.169260 16465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce6409f165134929b198d5b645f9a119. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:26.169250 16734 tablet_service.cc:1467] Processing CreateTablet for tablet ce6409f165134929b198d5b645f9a119 (DEFAULT_TABLE table=test-table [id=41cfe450a54040239425f359f49d3350]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:26.169358 16734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce6409f165134929b198d5b645f9a119. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:26.170053 16861 ts_tablet_manager.cc:1918] T f6f81f9570fd4f078644ce7c5022f908 P 720cc0480d0e4badabac18d7ff7e8bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:03:26.170127 16861 log.cc:1196] T f6f81f9570fd4f078644ce7c5022f908 P 720cc0480d0e4badabac18d7ff7e8bf4: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-1/wal/wals/f6f81f9570fd4f078644ce7c5022f908
I20260929 19:03:26.170387 16861 ts_tablet_manager.cc:1939] T f6f81f9570fd4f078644ce7c5022f908 P 720cc0480d0e4badabac18d7ff7e8bf4: Deleting consensus metadata
I20260929 19:03:26.170500 16860 tablet_replica.cc:330] T f6f81f9570fd4f078644ce7c5022f908 P 2c266e5fa9d94019907157fb874402b1: stopping tablet replica
I20260929 19:03:26.170607 16860 raft_consensus.cc:2229] T f6f81f9570fd4f078644ce7c5022f908 P 2c266e5fa9d94019907157fb874402b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:03:26.170611 16836 tablet_bootstrap.cc:492] T ce6409f165134929b198d5b645f9a119 P 9ea6b6f3d04e4b16aa63c8ad5dcee159: Bootstrap starting.
I20260929 19:03:26.170697 16860 raft_consensus.cc:2258] T f6f81f9570fd4f078644ce7c5022f908 P 2c266e5fa9d94019907157fb874402b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:03:26.171032 16837 tablet_bootstrap.cc:492] T ce6409f165134929b198d5b645f9a119 P 2c266e5fa9d94019907157fb874402b1: Bootstrap starting.
I20260929 19:03:26.171032 16836 tablet_bootstrap.cc:654] T ce6409f165134929b198d5b645f9a119 P 9ea6b6f3d04e4b16aa63c8ad5dcee159: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:26.171376 16837 tablet_bootstrap.cc:654] T ce6409f165134929b198d5b645f9a119 P 2c266e5fa9d94019907157fb874402b1: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:26.171437 16860 ts_tablet_manager.cc:1905] T f6f81f9570fd4f078644ce7c5022f908 P 2c266e5fa9d94019907157fb874402b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:03:26.171527 16859 ts_tablet_manager.cc:1905] T f6f81f9570fd4f078644ce7c5022f908 P 9ea6b6f3d04e4b16aa63c8ad5dcee159: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:03:26.171765 16312 catalog_manager.cc:4841] TS 720cc0480d0e4badabac18d7ff7e8bf4 (127.8.158.66:36373): tablet f6f81f9570fd4f078644ce7c5022f908 (table test-table [id=cdfd5e2042964b4ea9e4bbef2549c518]) successfully deleted
I20260929 19:03:26.171792 16605 tablet_service.cc:1467] Processing CreateTablet for tablet ce6409f165134929b198d5b645f9a119 (DEFAULT_TABLE table=test-table [id=41cfe450a54040239425f359f49d3350]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:26.171876 16605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce6409f165134929b198d5b645f9a119. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:26.172909 16860 ts_tablet_manager.cc:1918] T f6f81f9570fd4f078644ce7c5022f908 P 2c266e5fa9d94019907157fb874402b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:03:26.172967 16860 log.cc:1196] T f6f81f9570fd4f078644ce7c5022f908 P 2c266e5fa9d94019907157fb874402b1: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-0/wal/wals/f6f81f9570fd4f078644ce7c5022f908
I20260929 19:03:26.173137 16840 tablet_bootstrap.cc:492] T ce6409f165134929b198d5b645f9a119 P 720cc0480d0e4badabac18d7ff7e8bf4: Bootstrap starting.
I20260929 19:03:26.173218 16860 ts_tablet_manager.cc:1939] T f6f81f9570fd4f078644ce7c5022f908 P 2c266e5fa9d94019907157fb874402b1: Deleting consensus metadata
I20260929 19:03:26.173538 16840 tablet_bootstrap.cc:654] T ce6409f165134929b198d5b645f9a119 P 720cc0480d0e4badabac18d7ff7e8bf4: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:26.173599 16314 catalog_manager.cc:4841] TS 2c266e5fa9d94019907157fb874402b1 (127.8.158.65:34403): tablet f6f81f9570fd4f078644ce7c5022f908 (table test-table [id=cdfd5e2042964b4ea9e4bbef2549c518]) successfully deleted
I20260929 19:03:26.174154 16840 tablet_bootstrap.cc:492] T ce6409f165134929b198d5b645f9a119 P 720cc0480d0e4badabac18d7ff7e8bf4: No bootstrap required, opened a new log
I20260929 19:03:26.174212 16840 ts_tablet_manager.cc:1397] T ce6409f165134929b198d5b645f9a119 P 720cc0480d0e4badabac18d7ff7e8bf4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:26.174366 16840 raft_consensus.cc:348] T ce6409f165134929b198d5b645f9a119 P 720cc0480d0e4badabac18d7ff7e8bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } }
I20260929 19:03:26.174436 16840 raft_consensus.cc:374] T ce6409f165134929b198d5b645f9a119 P 720cc0480d0e4badabac18d7ff7e8bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:26.174465 16840 raft_consensus.cc:729] T ce6409f165134929b198d5b645f9a119 P 720cc0480d0e4badabac18d7ff7e8bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 720cc0480d0e4badabac18d7ff7e8bf4, State: Initialized, Role: FOLLOWER
I20260929 19:03:26.174504 16840 consensus_queue.cc:260] T ce6409f165134929b198d5b645f9a119 P 720cc0480d0e4badabac18d7ff7e8bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } }
I20260929 19:03:26.174597 16840 ts_tablet_manager.cc:1428] T ce6409f165134929b198d5b645f9a119 P 720cc0480d0e4badabac18d7ff7e8bf4: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 19:03:26.174842 16836 tablet_bootstrap.cc:492] T ce6409f165134929b198d5b645f9a119 P 9ea6b6f3d04e4b16aa63c8ad5dcee159: No bootstrap required, opened a new log
I20260929 19:03:26.174893 16836 ts_tablet_manager.cc:1397] T ce6409f165134929b198d5b645f9a119 P 9ea6b6f3d04e4b16aa63c8ad5dcee159: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260929 19:03:26.174990 16837 tablet_bootstrap.cc:492] T ce6409f165134929b198d5b645f9a119 P 2c266e5fa9d94019907157fb874402b1: No bootstrap required, opened a new log
I20260929 19:03:26.175027 16837 ts_tablet_manager.cc:1397] T ce6409f165134929b198d5b645f9a119 P 2c266e5fa9d94019907157fb874402b1: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260929 19:03:26.175066 16836 raft_consensus.cc:348] T ce6409f165134929b198d5b645f9a119 P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } }
I20260929 19:03:26.175129 16836 raft_consensus.cc:374] T ce6409f165134929b198d5b645f9a119 P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:26.175149 16836 raft_consensus.cc:729] T ce6409f165134929b198d5b645f9a119 P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ea6b6f3d04e4b16aa63c8ad5dcee159, State: Initialized, Role: FOLLOWER
I20260929 19:03:26.175168 16837 raft_consensus.cc:348] T ce6409f165134929b198d5b645f9a119 P 2c266e5fa9d94019907157fb874402b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } }
I20260929 19:03:26.175192 16836 consensus_queue.cc:260] T ce6409f165134929b198d5b645f9a119 P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } }
I20260929 19:03:26.175225 16837 raft_consensus.cc:374] T ce6409f165134929b198d5b645f9a119 P 2c266e5fa9d94019907157fb874402b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:26.175249 16837 raft_consensus.cc:729] T ce6409f165134929b198d5b645f9a119 P 2c266e5fa9d94019907157fb874402b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c266e5fa9d94019907157fb874402b1, State: Initialized, Role: FOLLOWER
I20260929 19:03:26.175271 16836 ts_tablet_manager.cc:1428] T ce6409f165134929b198d5b645f9a119 P 9ea6b6f3d04e4b16aa63c8ad5dcee159: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:26.175302 16837 consensus_queue.cc:260] T ce6409f165134929b198d5b645f9a119 P 2c266e5fa9d94019907157fb874402b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } }
I20260929 19:03:26.174885 16859 ts_tablet_manager.cc:1918] T f6f81f9570fd4f078644ce7c5022f908 P 9ea6b6f3d04e4b16aa63c8ad5dcee159: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:03:26.175611 16837 ts_tablet_manager.cc:1428] T ce6409f165134929b198d5b645f9a119 P 2c266e5fa9d94019907157fb874402b1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 19:03:26.175611 16859 log.cc:1196] T f6f81f9570fd4f078644ce7c5022f908 P 9ea6b6f3d04e4b16aa63c8ad5dcee159: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-2/wal/wals/f6f81f9570fd4f078644ce7c5022f908
I20260929 19:03:26.176041 16859 ts_tablet_manager.cc:1939] T f6f81f9570fd4f078644ce7c5022f908 P 9ea6b6f3d04e4b16aa63c8ad5dcee159: Deleting consensus metadata
I20260929 19:03:26.176414 16314 catalog_manager.cc:4841] TS 9ea6b6f3d04e4b16aa63c8ad5dcee159 (127.8.158.67:34541): tablet f6f81f9570fd4f078644ce7c5022f908 (table test-table [id=cdfd5e2042964b4ea9e4bbef2549c518]) successfully deleted
I20260929 19:03:26.346302 16843 raft_consensus.cc:482] T ce6409f165134929b198d5b645f9a119 P 2c266e5fa9d94019907157fb874402b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:03:26.346437 16843 raft_consensus.cc:504] T ce6409f165134929b198d5b645f9a119 P 2c266e5fa9d94019907157fb874402b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } }
I20260929 19:03:26.346800 16843 leader_election.cc:288] T ce6409f165134929b198d5b645f9a119 P 2c266e5fa9d94019907157fb874402b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 720cc0480d0e4badabac18d7ff7e8bf4 (127.8.158.66:36373), 9ea6b6f3d04e4b16aa63c8ad5dcee159 (127.8.158.67:34541)
I20260929 19:03:26.349618 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:26.347045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:57341 (local address 127.8.158.66:36373)
0929 19:03:26.347093 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:26.347096 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:26.347180 (+ 84us) server_negotiation.cc:407] Connection header received
0929 19:03:26.347249 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:26.347250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:26.347263 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:26.347314 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:26.347463 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.347929 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.348887 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.349253 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.349338 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:26.349350 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:26.349522 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 19:03:26.349560 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 19:03:26.349648 16835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:26.347094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:38377 (local address 127.8.158.67:34541)
0929 19:03:26.347135 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:26.347137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:26.347273 (+ 136us) server_negotiation.cc:407] Connection header received
0929 19:03:26.347335 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:26.347337 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:26.347346 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:26.347419 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:26.347569 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.348023 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.348917 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.349277 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.349359 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:26.349367 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:26.349535 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 19:03:26.349571 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 19:03:26.349912 16865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:26.346979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.66:36373 (local address 127.8.158.65:57341)
0929 19:03:26.347113 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:26.347124 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:26.347190 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:26.347317 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:26.347317 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:26.347319 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:26.347394 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:26.347395 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.348022 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.348023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:26.348795 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:26.348797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.349355 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.349356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:26.349431 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:26.349435 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:03:26.349502 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 19:03:26.349806 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":68,"threads_started":1}
I20260929 19:03:26.349944 16866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:26.347019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.67:34541 (local address 127.8.158.65:38377)
0929 19:03:26.347183 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:26.347200 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:26.347276 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:26.347421 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:26.347422 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:26.347423 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:26.347496 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:26.347498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.348123 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.348124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:26.348827 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:26.348828 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.349368 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.349369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:26.349445 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:26.349448 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:03:26.349528 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 19:03:26.349826 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":115,"threads_started":1}
I20260929 19:03:26.350204 16625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce6409f165134929b198d5b645f9a119" candidate_uuid: "2c266e5fa9d94019907157fb874402b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" is_pre_election: true
I20260929 19:03:26.350358 16625 raft_consensus.cc:2454] T ce6409f165134929b198d5b645f9a119 P 720cc0480d0e4badabac18d7ff7e8bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c266e5fa9d94019907157fb874402b1 in term 0.
I20260929 19:03:26.350348 16756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce6409f165134929b198d5b645f9a119" candidate_uuid: "2c266e5fa9d94019907157fb874402b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" is_pre_election: true
I20260929 19:03:26.350428 16756 raft_consensus.cc:2454] T ce6409f165134929b198d5b645f9a119 P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c266e5fa9d94019907157fb874402b1 in term 0.
I20260929 19:03:26.350569 16416 leader_election.cc:302] T ce6409f165134929b198d5b645f9a119 P 2c266e5fa9d94019907157fb874402b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c266e5fa9d94019907157fb874402b1, 720cc0480d0e4badabac18d7ff7e8bf4; no voters:
I20260929 19:03:26.350710 16843 raft_consensus.cc:2781] T ce6409f165134929b198d5b645f9a119 P 2c266e5fa9d94019907157fb874402b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:03:26.350750 16843 raft_consensus.cc:482] T ce6409f165134929b198d5b645f9a119 P 2c266e5fa9d94019907157fb874402b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:03:26.350771 16843 raft_consensus.cc:3037] T ce6409f165134929b198d5b645f9a119 P 2c266e5fa9d94019907157fb874402b1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:26.351377 16843 raft_consensus.cc:504] T ce6409f165134929b198d5b645f9a119 P 2c266e5fa9d94019907157fb874402b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } }
I20260929 19:03:26.351509 16843 leader_election.cc:288] T ce6409f165134929b198d5b645f9a119 P 2c266e5fa9d94019907157fb874402b1 [CANDIDATE]: Term 1 election: Requested vote from peers 720cc0480d0e4badabac18d7ff7e8bf4 (127.8.158.66:36373), 9ea6b6f3d04e4b16aa63c8ad5dcee159 (127.8.158.67:34541)
I20260929 19:03:26.351666 16756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce6409f165134929b198d5b645f9a119" candidate_uuid: "2c266e5fa9d94019907157fb874402b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159"
I20260929 19:03:26.351723 16756 raft_consensus.cc:3037] T ce6409f165134929b198d5b645f9a119 P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:26.352219 16756 raft_consensus.cc:2454] T ce6409f165134929b198d5b645f9a119 P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c266e5fa9d94019907157fb874402b1 in term 1.
I20260929 19:03:26.352301 16625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce6409f165134929b198d5b645f9a119" candidate_uuid: "2c266e5fa9d94019907157fb874402b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "720cc0480d0e4badabac18d7ff7e8bf4"
I20260929 19:03:26.352349 16625 raft_consensus.cc:3037] T ce6409f165134929b198d5b645f9a119 P 720cc0480d0e4badabac18d7ff7e8bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:26.352504 16418 leader_election.cc:302] T ce6409f165134929b198d5b645f9a119 P 2c266e5fa9d94019907157fb874402b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c266e5fa9d94019907157fb874402b1, 9ea6b6f3d04e4b16aa63c8ad5dcee159; no voters:
I20260929 19:03:26.352653 16843 raft_consensus.cc:2781] T ce6409f165134929b198d5b645f9a119 P 2c266e5fa9d94019907157fb874402b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:03:26.352794 16843 raft_consensus.cc:686] T ce6409f165134929b198d5b645f9a119 P 2c266e5fa9d94019907157fb874402b1 [term 1 LEADER]: Becoming Leader. State: Replica: 2c266e5fa9d94019907157fb874402b1, State: Running, Role: LEADER
I20260929 19:03:26.352885 16843 consensus_queue.cc:237] T ce6409f165134929b198d5b645f9a119 P 2c266e5fa9d94019907157fb874402b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } }
I20260929 19:03:26.353176 16625 raft_consensus.cc:2454] T ce6409f165134929b198d5b645f9a119 P 720cc0480d0e4badabac18d7ff7e8bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c266e5fa9d94019907157fb874402b1 in term 1.
I20260929 19:03:26.353549 16327 catalog_manager.cc:5495] T ce6409f165134929b198d5b645f9a119 P 2c266e5fa9d94019907157fb874402b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c266e5fa9d94019907157fb874402b1 (127.8.158.65). New cstate: current_term: 1 leader_uuid: "2c266e5fa9d94019907157fb874402b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } health_report { overall_health: UNKNOWN } } }
Sep 29 19:03:26 dist-test-slave-nmsz krb5kdc[16283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708605, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.158.126@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM
I20260929 19:03:26.418424 16874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:26.411265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:43799 (local address 127.0.0.1:57190)
0929 19:03:26.411469 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:26.411486 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:26.411583 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:26.411744 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:26.411745 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:26.412040 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:26.412078 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:26.412080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.412712 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.412713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:26.413252 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:26.413254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.413830 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.413832 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:26.413912 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:26.414612 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:26.414618 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:26.416021 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:26.417481 (+ 1460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:26.417486 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:26.417488 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:26.417698 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:26.417905 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:26.417906 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:26.417906 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:26.417959 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:26.418162 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:26.418167 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:26.418203 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 19:03:26.418252 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 19:03:26.418354 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":108,"threads_started":1}
I20260929 19:03:26.418490 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:26.411369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57190 (local address 127.8.158.126:43799)
0929 19:03:26.411448 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:26.411450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:26.411565 (+ 115us) server_negotiation.cc:407] Connection header received
0929 19:03:26.411653 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:26.411654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:26.411664 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:26.411741 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:26.412161 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.412634 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.413411 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.413710 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.413808 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:26.416110 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:26.416111 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:26.416111 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:26.416117 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:26.417403 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:26.417785 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:26.417786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:26.417787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:26.417842 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:26.418012 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:26.418013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:26.418013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:26.418111 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:26.418161 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:26.418269 (+ 108us) server_negotiation.cc:299] Negotiation successful
0929 19:03:26.418416 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":53}
I20260929 19:03:26.420212 16327 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:57190:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 19:03:26.420399 16327 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:03:26.421800 16734 tablet_service.cc:1467] Processing CreateTablet for tablet b6ab192d002c454d8e3f2425e6c52365 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a4e181f7d9e647a1a6c8fd525e27f0c6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 19:03:26.421792 16605 tablet_service.cc:1467] Processing CreateTablet for tablet b6ab192d002c454d8e3f2425e6c52365 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a4e181f7d9e647a1a6c8fd525e27f0c6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 19:03:26.421944 16605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6ab192d002c454d8e3f2425e6c52365. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:26.421944 16734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6ab192d002c454d8e3f2425e6c52365. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:26.423385 16836 tablet_bootstrap.cc:492] T b6ab192d002c454d8e3f2425e6c52365 P 9ea6b6f3d04e4b16aa63c8ad5dcee159: Bootstrap starting.
I20260929 19:03:26.423493 16465 tablet_service.cc:1467] Processing CreateTablet for tablet b6ab192d002c454d8e3f2425e6c52365 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a4e181f7d9e647a1a6c8fd525e27f0c6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 19:03:26.423633 16465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6ab192d002c454d8e3f2425e6c52365. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:26.423818 16836 tablet_bootstrap.cc:654] T b6ab192d002c454d8e3f2425e6c52365 P 9ea6b6f3d04e4b16aa63c8ad5dcee159: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:26.424109 16840 tablet_bootstrap.cc:492] T b6ab192d002c454d8e3f2425e6c52365 P 720cc0480d0e4badabac18d7ff7e8bf4: Bootstrap starting.
I20260929 19:03:26.424392 16836 tablet_bootstrap.cc:492] T b6ab192d002c454d8e3f2425e6c52365 P 9ea6b6f3d04e4b16aa63c8ad5dcee159: No bootstrap required, opened a new log
I20260929 19:03:26.424412 16840 tablet_bootstrap.cc:654] T b6ab192d002c454d8e3f2425e6c52365 P 720cc0480d0e4badabac18d7ff7e8bf4: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:26.424448 16836 ts_tablet_manager.cc:1397] T b6ab192d002c454d8e3f2425e6c52365 P 9ea6b6f3d04e4b16aa63c8ad5dcee159: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:26.424593 16836 raft_consensus.cc:348] T b6ab192d002c454d8e3f2425e6c52365 P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } }
I20260929 19:03:26.424660 16836 raft_consensus.cc:374] T b6ab192d002c454d8e3f2425e6c52365 P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:26.424680 16836 raft_consensus.cc:729] T b6ab192d002c454d8e3f2425e6c52365 P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ea6b6f3d04e4b16aa63c8ad5dcee159, State: Initialized, Role: FOLLOWER
I20260929 19:03:26.424721 16836 consensus_queue.cc:260] T b6ab192d002c454d8e3f2425e6c52365 P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } }
I20260929 19:03:26.424829 16836 ts_tablet_manager.cc:1428] T b6ab192d002c454d8e3f2425e6c52365 P 9ea6b6f3d04e4b16aa63c8ad5dcee159: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:26.424880 16842 tablet_replica.cc:439] T b6ab192d002c454d8e3f2425e6c52365 P 9ea6b6f3d04e4b16aa63c8ad5dcee159: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } } }
I20260929 19:03:26.424990 16842 tablet_replica.cc:442] T b6ab192d002c454d8e3f2425e6c52365 P 9ea6b6f3d04e4b16aa63c8ad5dcee159: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 19:03:26.425086 16837 tablet_bootstrap.cc:492] T b6ab192d002c454d8e3f2425e6c52365 P 2c266e5fa9d94019907157fb874402b1: Bootstrap starting.
I20260929 19:03:26.425424 16840 tablet_bootstrap.cc:492] T b6ab192d002c454d8e3f2425e6c52365 P 720cc0480d0e4badabac18d7ff7e8bf4: No bootstrap required, opened a new log
I20260929 19:03:26.425470 16840 ts_tablet_manager.cc:1397] T b6ab192d002c454d8e3f2425e6c52365 P 720cc0480d0e4badabac18d7ff7e8bf4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:26.425490 16837 tablet_bootstrap.cc:654] T b6ab192d002c454d8e3f2425e6c52365 P 2c266e5fa9d94019907157fb874402b1: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:26.425647 16840 raft_consensus.cc:348] T b6ab192d002c454d8e3f2425e6c52365 P 720cc0480d0e4badabac18d7ff7e8bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } }
I20260929 19:03:26.425697 16840 raft_consensus.cc:374] T b6ab192d002c454d8e3f2425e6c52365 P 720cc0480d0e4badabac18d7ff7e8bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:26.425710 16840 raft_consensus.cc:729] T b6ab192d002c454d8e3f2425e6c52365 P 720cc0480d0e4badabac18d7ff7e8bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 720cc0480d0e4badabac18d7ff7e8bf4, State: Initialized, Role: FOLLOWER
I20260929 19:03:26.425765 16840 consensus_queue.cc:260] T b6ab192d002c454d8e3f2425e6c52365 P 720cc0480d0e4badabac18d7ff7e8bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } }
I20260929 19:03:26.425850 16840 ts_tablet_manager.cc:1428] T b6ab192d002c454d8e3f2425e6c52365 P 720cc0480d0e4badabac18d7ff7e8bf4: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 19:03:26.425866 16856 tablet_replica.cc:439] T b6ab192d002c454d8e3f2425e6c52365 P 720cc0480d0e4badabac18d7ff7e8bf4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } } }
I20260929 19:03:26.425935 16856 tablet_replica.cc:442] T b6ab192d002c454d8e3f2425e6c52365 P 720cc0480d0e4badabac18d7ff7e8bf4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 19:03:26.426168 16837 tablet_bootstrap.cc:492] T b6ab192d002c454d8e3f2425e6c52365 P 2c266e5fa9d94019907157fb874402b1: No bootstrap required, opened a new log
I20260929 19:03:26.426219 16837 ts_tablet_manager.cc:1397] T b6ab192d002c454d8e3f2425e6c52365 P 2c266e5fa9d94019907157fb874402b1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:26.426378 16837 raft_consensus.cc:348] T b6ab192d002c454d8e3f2425e6c52365 P 2c266e5fa9d94019907157fb874402b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } }
I20260929 19:03:26.426469 16837 raft_consensus.cc:374] T b6ab192d002c454d8e3f2425e6c52365 P 2c266e5fa9d94019907157fb874402b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:26.426504 16837 raft_consensus.cc:729] T b6ab192d002c454d8e3f2425e6c52365 P 2c266e5fa9d94019907157fb874402b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c266e5fa9d94019907157fb874402b1, State: Initialized, Role: FOLLOWER
I20260929 19:03:26.426555 16837 consensus_queue.cc:260] T b6ab192d002c454d8e3f2425e6c52365 P 2c266e5fa9d94019907157fb874402b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } }
I20260929 19:03:26.426652 16837 ts_tablet_manager.cc:1428] T b6ab192d002c454d8e3f2425e6c52365 P 2c266e5fa9d94019907157fb874402b1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:26.426684 16843 tablet_replica.cc:439] T b6ab192d002c454d8e3f2425e6c52365 P 2c266e5fa9d94019907157fb874402b1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } } }
I20260929 19:03:26.426771 16843 tablet_replica.cc:442] T b6ab192d002c454d8e3f2425e6c52365 P 2c266e5fa9d94019907157fb874402b1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 19:03:26.434015 16856 raft_consensus.cc:482] T b6ab192d002c454d8e3f2425e6c52365 P 720cc0480d0e4badabac18d7ff7e8bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:03:26.434103 16856 raft_consensus.cc:504] T b6ab192d002c454d8e3f2425e6c52365 P 720cc0480d0e4badabac18d7ff7e8bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } }
I20260929 19:03:26.434237 16856 leader_election.cc:288] T b6ab192d002c454d8e3f2425e6c52365 P 720cc0480d0e4badabac18d7ff7e8bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c266e5fa9d94019907157fb874402b1 (127.8.158.65:34403), 9ea6b6f3d04e4b16aa63c8ad5dcee159 (127.8.158.67:34541)
I20260929 19:03:26.434407 16485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ab192d002c454d8e3f2425e6c52365" candidate_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c266e5fa9d94019907157fb874402b1" is_pre_election: true
I20260929 19:03:26.434478 16485 raft_consensus.cc:2454] T b6ab192d002c454d8e3f2425e6c52365 P 2c266e5fa9d94019907157fb874402b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 720cc0480d0e4badabac18d7ff7e8bf4 in term 0.
I20260929 19:03:26.434464 16756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ab192d002c454d8e3f2425e6c52365" candidate_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" is_pre_election: true
I20260929 19:03:26.434541 16756 raft_consensus.cc:2454] T b6ab192d002c454d8e3f2425e6c52365 P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 720cc0480d0e4badabac18d7ff7e8bf4 in term 0.
I20260929 19:03:26.434648 16558 leader_election.cc:302] T b6ab192d002c454d8e3f2425e6c52365 P 720cc0480d0e4badabac18d7ff7e8bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c266e5fa9d94019907157fb874402b1, 720cc0480d0e4badabac18d7ff7e8bf4; no voters:
I20260929 19:03:26.434744 16856 raft_consensus.cc:2781] T b6ab192d002c454d8e3f2425e6c52365 P 720cc0480d0e4badabac18d7ff7e8bf4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:03:26.434790 16856 raft_consensus.cc:482] T b6ab192d002c454d8e3f2425e6c52365 P 720cc0480d0e4badabac18d7ff7e8bf4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:03:26.434820 16856 raft_consensus.cc:3037] T b6ab192d002c454d8e3f2425e6c52365 P 720cc0480d0e4badabac18d7ff7e8bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:26.435393 16856 raft_consensus.cc:504] T b6ab192d002c454d8e3f2425e6c52365 P 720cc0480d0e4badabac18d7ff7e8bf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } }
I20260929 19:03:26.435515 16856 leader_election.cc:288] T b6ab192d002c454d8e3f2425e6c52365 P 720cc0480d0e4badabac18d7ff7e8bf4 [CANDIDATE]: Term 1 election: Requested vote from peers 2c266e5fa9d94019907157fb874402b1 (127.8.158.65:34403), 9ea6b6f3d04e4b16aa63c8ad5dcee159 (127.8.158.67:34541)
I20260929 19:03:26.435724 16756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ab192d002c454d8e3f2425e6c52365" candidate_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159"
I20260929 19:03:26.435729 16485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ab192d002c454d8e3f2425e6c52365" candidate_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c266e5fa9d94019907157fb874402b1"
I20260929 19:03:26.435804 16756 raft_consensus.cc:3037] T b6ab192d002c454d8e3f2425e6c52365 P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:26.435804 16485 raft_consensus.cc:3037] T b6ab192d002c454d8e3f2425e6c52365 P 2c266e5fa9d94019907157fb874402b1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:26.436467 16756 raft_consensus.cc:2454] T b6ab192d002c454d8e3f2425e6c52365 P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 720cc0480d0e4badabac18d7ff7e8bf4 in term 1.
I20260929 19:03:26.436467 16485 raft_consensus.cc:2454] T b6ab192d002c454d8e3f2425e6c52365 P 2c266e5fa9d94019907157fb874402b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 720cc0480d0e4badabac18d7ff7e8bf4 in term 1.
I20260929 19:03:26.436637 16558 leader_election.cc:302] T b6ab192d002c454d8e3f2425e6c52365 P 720cc0480d0e4badabac18d7ff7e8bf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c266e5fa9d94019907157fb874402b1, 720cc0480d0e4badabac18d7ff7e8bf4; no voters:
I20260929 19:03:26.436723 16856 raft_consensus.cc:2781] T b6ab192d002c454d8e3f2425e6c52365 P 720cc0480d0e4badabac18d7ff7e8bf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:03:26.436777 16856 raft_consensus.cc:686] T b6ab192d002c454d8e3f2425e6c52365 P 720cc0480d0e4badabac18d7ff7e8bf4 [term 1 LEADER]: Becoming Leader. State: Replica: 720cc0480d0e4badabac18d7ff7e8bf4, State: Running, Role: LEADER
I20260929 19:03:26.436826 16856 consensus_queue.cc:237] T b6ab192d002c454d8e3f2425e6c52365 P 720cc0480d0e4badabac18d7ff7e8bf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } }
I20260929 19:03:26.437062 16846 tablet_replica.cc:439] T b6ab192d002c454d8e3f2425e6c52365 P 720cc0480d0e4badabac18d7ff7e8bf4: TxnStatusTablet state changed. Reason: New leader 720cc0480d0e4badabac18d7ff7e8bf4. Latest consensus state: current_term: 1 leader_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } } }
I20260929 19:03:26.437153 16846 tablet_replica.cc:442] T b6ab192d002c454d8e3f2425e6c52365 P 720cc0480d0e4badabac18d7ff7e8bf4: This TxnStatusTablet replica's current role is: LEADER
I20260929 19:03:26.437317 16879 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 19:03:26.437325 16328 catalog_manager.cc:5495] T b6ab192d002c454d8e3f2425e6c52365 P 720cc0480d0e4badabac18d7ff7e8bf4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 720cc0480d0e4badabac18d7ff7e8bf4 (127.8.158.66). New cstate: current_term: 1 leader_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } health_report { overall_health: UNKNOWN } } }
I20260929 19:03:26.437440 16879 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 19:03:26.442560 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:26.440086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49522 (local address 127.8.158.66:36373)
0929 19:03:26.440122 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:26.440124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:26.440149 (+ 25us) server_negotiation.cc:407] Connection header received
0929 19:03:26.440227 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:26.440227 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:26.440234 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:26.440280 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:03:26.440389 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.440832 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.441881 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.442067 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.442126 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:26.442301 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:03:26.442369 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:03:26.442420 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:26.442499 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 19:03:26.442517 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 19:03:26.442783 16874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:26.440028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.66:36373 (local address 127.0.0.1:49522)
0929 19:03:26.440105 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:26.440116 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:26.440167 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:26.440287 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:26.440288 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:26.440289 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 19:03:26.440332 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:26.440334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.440902 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.440903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:26.441791 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:26.441794 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.442142 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.442143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:26.442199 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:26.442215 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 19:03:26.442430 (+ 215us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 19:03:26.442438 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 19:03:26.442504 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 19:03:26.442550 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 19:03:26.443569 16756 raft_consensus.cc:1261] T b6ab192d002c454d8e3f2425e6c52365 P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [term 1 FOLLOWER]: Refusing update from remote peer 720cc0480d0e4badabac18d7ff7e8bf4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:03:26.443611 16485 raft_consensus.cc:1261] T b6ab192d002c454d8e3f2425e6c52365 P 2c266e5fa9d94019907157fb874402b1 [term 1 FOLLOWER]: Refusing update from remote peer 720cc0480d0e4badabac18d7ff7e8bf4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:03:26.443775 16846 consensus_queue.cc:1035] T b6ab192d002c454d8e3f2425e6c52365 P 720cc0480d0e4badabac18d7ff7e8bf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:03:26.443853 16856 consensus_queue.cc:1035] T b6ab192d002c454d8e3f2425e6c52365 P 720cc0480d0e4badabac18d7ff7e8bf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:03:26.444378 16843 tablet_replica.cc:439] T b6ab192d002c454d8e3f2425e6c52365 P 2c266e5fa9d94019907157fb874402b1: TxnStatusTablet state changed. Reason: New leader 720cc0480d0e4badabac18d7ff7e8bf4. Latest consensus state: current_term: 1 leader_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } } }
I20260929 19:03:26.444454 16843 tablet_replica.cc:442] T b6ab192d002c454d8e3f2425e6c52365 P 2c266e5fa9d94019907157fb874402b1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 19:03:26.444439 16842 tablet_replica.cc:439] T b6ab192d002c454d8e3f2425e6c52365 P 9ea6b6f3d04e4b16aa63c8ad5dcee159: TxnStatusTablet state changed. Reason: New leader 720cc0480d0e4badabac18d7ff7e8bf4. Latest consensus state: current_term: 1 leader_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } } }
I20260929 19:03:26.444540 16842 tablet_replica.cc:442] T b6ab192d002c454d8e3f2425e6c52365 P 9ea6b6f3d04e4b16aa63c8ad5dcee159: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 19:03:26.445031 16856 tablet_replica.cc:439] T b6ab192d002c454d8e3f2425e6c52365 P 720cc0480d0e4badabac18d7ff7e8bf4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } } }
I20260929 19:03:26.445109 16856 tablet_replica.cc:442] T b6ab192d002c454d8e3f2425e6c52365 P 720cc0480d0e4badabac18d7ff7e8bf4: This TxnStatusTablet replica's current role is: LEADER
I20260929 19:03:26.445344 16842 tablet_replica.cc:439] T b6ab192d002c454d8e3f2425e6c52365 P 9ea6b6f3d04e4b16aa63c8ad5dcee159: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } } }
I20260929 19:03:26.445430 16842 tablet_replica.cc:442] T b6ab192d002c454d8e3f2425e6c52365 P 9ea6b6f3d04e4b16aa63c8ad5dcee159: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 19:03:26.445923 16856 tablet_replica.cc:439] T b6ab192d002c454d8e3f2425e6c52365 P 720cc0480d0e4badabac18d7ff7e8bf4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } } }
I20260929 19:03:26.445994 16856 tablet_replica.cc:442] T b6ab192d002c454d8e3f2425e6c52365 P 720cc0480d0e4badabac18d7ff7e8bf4: This TxnStatusTablet replica's current role is: LEADER
I20260929 19:03:26.446228 16843 tablet_replica.cc:439] T b6ab192d002c454d8e3f2425e6c52365 P 2c266e5fa9d94019907157fb874402b1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } } }
I20260929 19:03:26.446305 16843 tablet_replica.cc:442] T b6ab192d002c454d8e3f2425e6c52365 P 2c266e5fa9d94019907157fb874402b1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 19:03:26.449143 16834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:26.446644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41512 (local address 127.8.158.65:34403)
0929 19:03:26.446684 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:26.446686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:26.446695 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:03:26.446726 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:26.446727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:26.446745 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:26.446809 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:03:26.446929 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.447428 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.448455 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.448652 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.448709 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:26.448858 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:03:26.448942 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:03:26.448991 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:26.449055 (+ 64us) server_negotiation.cc:299] Negotiation successful
0929 19:03:26.449071 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 19:03:26.453698 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:26.451323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:41397 (local address 127.8.158.66:36373)
0929 19:03:26.451363 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:26.451364 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:26.451462 (+ 98us) server_negotiation.cc:407] Connection header received
0929 19:03:26.451542 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:26.451542 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:26.451551 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:26.451593 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:03:26.451717 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.452118 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.453163 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.453325 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.453373 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:26.453546 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:03:26.453566 (+ 20us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:03:26.453602 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:26.453646 (+ 44us) server_negotiation.cc:299] Negotiation successful
0929 19:03:26.453659 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 19:03:26.453789 16886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:26.451273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.66:36373 (local address 127.8.158.65:41397)
0929 19:03:26.451406 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:26.451416 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:26.451477 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:26.451597 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:26.451597 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:26.451598 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 19:03:26.451654 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:26.451655 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.452184 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.452185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:26.453085 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:26.453087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.453391 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.453392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:26.453453 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:26.453466 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 19:03:26.453603 (+ 137us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 19:03:26.453604 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 19:03:26.453645 (+ 41us) client_negotiation.cc:240] Negotiation successful
0929 19:03:26.453682 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":71,"threads_started":1}
I20260929 19:03:26.455698 16756 raft_consensus.cc:1261] T ce6409f165134929b198d5b645f9a119 P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [term 1 FOLLOWER]: Refusing update from remote peer 2c266e5fa9d94019907157fb874402b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:03:26.455735 16625 raft_consensus.cc:1261] T ce6409f165134929b198d5b645f9a119 P 720cc0480d0e4badabac18d7ff7e8bf4 [term 1 FOLLOWER]: Refusing update from remote peer 2c266e5fa9d94019907157fb874402b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:03:26.455924 16867 consensus_queue.cc:1035] T ce6409f165134929b198d5b645f9a119 P 2c266e5fa9d94019907157fb874402b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:03:26.456014 16843 consensus_queue.cc:1035] T ce6409f165134929b198d5b645f9a119 P 2c266e5fa9d94019907157fb874402b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:03:26.464219 16834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:26.461779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:53949 (local address 127.8.158.65:34403)
0929 19:03:26.461814 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:26.461816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:26.461928 (+ 112us) server_negotiation.cc:407] Connection header received
0929 19:03:26.461994 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:26.461995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:26.462002 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:26.462040 (+ 38us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:03:26.462141 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.462623 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.463682 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.463848 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.463897 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:26.464057 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:03:26.464077 (+ 20us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:03:26.464114 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:26.464166 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 19:03:26.464179 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 19:03:26.464272 16890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:26.461741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:34403 (local address 127.8.158.66:53949)
0929 19:03:26.461872 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:26.461883 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:26.461941 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:26.462051 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:26.462052 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:26.462053 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 19:03:26.462089 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:26.462091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.462692 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.462692 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:26.463599 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:26.463601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.463914 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.463915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:26.463970 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:26.463983 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 19:03:26.464115 (+ 132us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 19:03:26.464116 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 19:03:26.464164 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 19:03:26.464204 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":74,"threads_started":1}
I20260929 19:03:26.472978 16327 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:03:26.473063 16327 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:03:26.473620 16327 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 61f4a227b7324cb584378135dfc9874a: Sending DeleteTablet for 3 replicas of tablet ce6409f165134929b198d5b645f9a119
I20260929 19:03:26.473834 16465 tablet_service.cc:1514] Processing DeleteTablet for tablet ce6409f165134929b198d5b645f9a119 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:26 UTC) from {username='oryx', principal='oryx/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:41502
I20260929 19:03:26.473865 16734 tablet_service.cc:1514] Processing DeleteTablet for tablet ce6409f165134929b198d5b645f9a119 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:26 UTC) from {username='oryx', principal='oryx/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:34246
I20260929 19:03:26.473914 16860 tablet_replica.cc:330] T ce6409f165134929b198d5b645f9a119 P 2c266e5fa9d94019907157fb874402b1: stopping tablet replica
I20260929 19:03:26.473943 16859 tablet_replica.cc:330] T ce6409f165134929b198d5b645f9a119 P 9ea6b6f3d04e4b16aa63c8ad5dcee159: stopping tablet replica
I20260929 19:03:26.473953 16860 raft_consensus.cc:2229] T ce6409f165134929b198d5b645f9a119 P 2c266e5fa9d94019907157fb874402b1 [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:03:26.473984 16859 raft_consensus.cc:2229] T ce6409f165134929b198d5b645f9a119 P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:03:26.474013 16859 raft_consensus.cc:2258] T ce6409f165134929b198d5b645f9a119 P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:03:26.474018 16860 raft_consensus.cc:2258] T ce6409f165134929b198d5b645f9a119 P 2c266e5fa9d94019907157fb874402b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:03:26.474299 16605 tablet_service.cc:1514] Processing DeleteTablet for tablet ce6409f165134929b198d5b645f9a119 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:26 UTC) from {username='oryx', principal='oryx/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:49494
I20260929 19:03:26.474408 16861 tablet_replica.cc:330] T ce6409f165134929b198d5b645f9a119 P 720cc0480d0e4badabac18d7ff7e8bf4: stopping tablet replica
I20260929 19:03:26.474450 16861 raft_consensus.cc:2229] T ce6409f165134929b198d5b645f9a119 P 720cc0480d0e4badabac18d7ff7e8bf4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:03:26.474455 16860 ts_tablet_manager.cc:1905] T ce6409f165134929b198d5b645f9a119 P 2c266e5fa9d94019907157fb874402b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:03:26.474480 16861 raft_consensus.cc:2258] T ce6409f165134929b198d5b645f9a119 P 720cc0480d0e4badabac18d7ff7e8bf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:03:26.476227 16861 ts_tablet_manager.cc:1905] T ce6409f165134929b198d5b645f9a119 P 720cc0480d0e4badabac18d7ff7e8bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:03:26.477349 16859 ts_tablet_manager.cc:1905] T ce6409f165134929b198d5b645f9a119 P 9ea6b6f3d04e4b16aa63c8ad5dcee159: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:03:26.477556 16861 ts_tablet_manager.cc:1918] T ce6409f165134929b198d5b645f9a119 P 720cc0480d0e4badabac18d7ff7e8bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 19:03:26.477603 16861 log.cc:1196] T ce6409f165134929b198d5b645f9a119 P 720cc0480d0e4badabac18d7ff7e8bf4: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-1/wal/wals/ce6409f165134929b198d5b645f9a119
I20260929 19:03:26.477815 16861 ts_tablet_manager.cc:1939] T ce6409f165134929b198d5b645f9a119 P 720cc0480d0e4badabac18d7ff7e8bf4: Deleting consensus metadata
I20260929 19:03:26.478103 16860 ts_tablet_manager.cc:1918] T ce6409f165134929b198d5b645f9a119 P 2c266e5fa9d94019907157fb874402b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 19:03:26.478152 16860 log.cc:1196] T ce6409f165134929b198d5b645f9a119 P 2c266e5fa9d94019907157fb874402b1: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-0/wal/wals/ce6409f165134929b198d5b645f9a119
I20260929 19:03:26.478368 16860 ts_tablet_manager.cc:1939] T ce6409f165134929b198d5b645f9a119 P 2c266e5fa9d94019907157fb874402b1: Deleting consensus metadata
I20260929 19:03:26.478683 16314 catalog_manager.cc:4841] TS 2c266e5fa9d94019907157fb874402b1 (127.8.158.65:34403): tablet ce6409f165134929b198d5b645f9a119 (table test-table [id=41cfe450a54040239425f359f49d3350]) successfully deleted
I20260929 19:03:26.478792 16859 ts_tablet_manager.cc:1918] T ce6409f165134929b198d5b645f9a119 P 9ea6b6f3d04e4b16aa63c8ad5dcee159: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 19:03:26.478852 16859 log.cc:1196] T ce6409f165134929b198d5b645f9a119 P 9ea6b6f3d04e4b16aa63c8ad5dcee159: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-2/wal/wals/ce6409f165134929b198d5b645f9a119
I20260929 19:03:26.479061 16859 ts_tablet_manager.cc:1939] T ce6409f165134929b198d5b645f9a119 P 9ea6b6f3d04e4b16aa63c8ad5dcee159: Deleting consensus metadata
I20260929 19:03:26.479382 16312 catalog_manager.cc:4841] TS 720cc0480d0e4badabac18d7ff7e8bf4 (127.8.158.66:36373): tablet ce6409f165134929b198d5b645f9a119 (table test-table [id=41cfe450a54040239425f359f49d3350]) successfully deleted
I20260929 19:03:26.479379 16314 catalog_manager.cc:4841] TS 9ea6b6f3d04e4b16aa63c8ad5dcee159 (127.8.158.67:34541): tablet ce6409f165134929b198d5b645f9a119 (table test-table [id=41cfe450a54040239425f359f49d3350]) successfully deleted
Sep 29 19:03:26 dist-test-slave-nmsz krb5kdc[16283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 19:03:26 dist-test-slave-nmsz krb5kdc[16283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:26 dist-test-slave-nmsz krb5kdc[16283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 19:03:26.508000 16378 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:26.501070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57200 (local address 127.8.158.126:43799)
0929 19:03:26.501115 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:26.501117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:26.501279 (+ 162us) server_negotiation.cc:407] Connection header received
0929 19:03:26.501332 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:26.501333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:26.501345 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:26.501406 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:26.502150 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.502991 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.504181 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.504478 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.504545 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:26.507936 (+ 3391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57200: BlockingRecv error: recv got EOF from 127.0.0.1:57200 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 29 19:03:26 dist-test-slave-nmsz krb5kdc[16283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM
I20260929 19:03:26.518925 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:26.508657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57202 (local address 127.8.158.126:43799)
0929 19:03:26.508699 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:26.508701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:26.510506 (+ 1805us) server_negotiation.cc:407] Connection header received
0929 19:03:26.510543 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:26.510544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:26.510555 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:26.510627 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:26.511205 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.511972 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.512967 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.513258 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.513329 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:26.515928 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:26.515930 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:26.515930 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:26.515937 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:26.517711 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:26.518096 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:26.518097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:26.518097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:26.518159 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:26.518393 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:26.518394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:26.518394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:26.518520 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:26.518618 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:26.518758 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 19:03:26.518831 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 19:03:26.520396 16327 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:03:26.520555 16327 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:03:26.521876 16734 tablet_service.cc:1467] Processing CreateTablet for tablet 12e679a662f746faa893b0d9edabbb29 (DEFAULT_TABLE table=test-table [id=d886f776ae7544ad9897b502d39b66ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:26.521950 16465 tablet_service.cc:1467] Processing CreateTablet for tablet 12e679a662f746faa893b0d9edabbb29 (DEFAULT_TABLE table=test-table [id=d886f776ae7544ad9897b502d39b66ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:26.521999 16734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12e679a662f746faa893b0d9edabbb29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:26.522070 16465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12e679a662f746faa893b0d9edabbb29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:26.523399 16836 tablet_bootstrap.cc:492] T 12e679a662f746faa893b0d9edabbb29 P 9ea6b6f3d04e4b16aa63c8ad5dcee159: Bootstrap starting.
I20260929 19:03:26.523433 16605 tablet_service.cc:1467] Processing CreateTablet for tablet 12e679a662f746faa893b0d9edabbb29 (DEFAULT_TABLE table=test-table [id=d886f776ae7544ad9897b502d39b66ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:26.523540 16605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12e679a662f746faa893b0d9edabbb29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:26.523803 16836 tablet_bootstrap.cc:654] T 12e679a662f746faa893b0d9edabbb29 P 9ea6b6f3d04e4b16aa63c8ad5dcee159: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:26.524782 16840 tablet_bootstrap.cc:492] T 12e679a662f746faa893b0d9edabbb29 P 720cc0480d0e4badabac18d7ff7e8bf4: Bootstrap starting.
I20260929 19:03:26.524806 16837 tablet_bootstrap.cc:492] T 12e679a662f746faa893b0d9edabbb29 P 2c266e5fa9d94019907157fb874402b1: Bootstrap starting.
I20260929 19:03:26.525058 16840 tablet_bootstrap.cc:654] T 12e679a662f746faa893b0d9edabbb29 P 720cc0480d0e4badabac18d7ff7e8bf4: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:26.525197 16837 tablet_bootstrap.cc:654] T 12e679a662f746faa893b0d9edabbb29 P 2c266e5fa9d94019907157fb874402b1: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:26.525825 16837 tablet_bootstrap.cc:492] T 12e679a662f746faa893b0d9edabbb29 P 2c266e5fa9d94019907157fb874402b1: No bootstrap required, opened a new log
I20260929 19:03:26.525871 16837 ts_tablet_manager.cc:1397] T 12e679a662f746faa893b0d9edabbb29 P 2c266e5fa9d94019907157fb874402b1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:26.526048 16836 tablet_bootstrap.cc:492] T 12e679a662f746faa893b0d9edabbb29 P 9ea6b6f3d04e4b16aa63c8ad5dcee159: No bootstrap required, opened a new log
I20260929 19:03:26.526006 16837 raft_consensus.cc:348] T 12e679a662f746faa893b0d9edabbb29 P 2c266e5fa9d94019907157fb874402b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } }
I20260929 19:03:26.526093 16837 raft_consensus.cc:374] T 12e679a662f746faa893b0d9edabbb29 P 2c266e5fa9d94019907157fb874402b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:26.526091 16836 ts_tablet_manager.cc:1397] T 12e679a662f746faa893b0d9edabbb29 P 9ea6b6f3d04e4b16aa63c8ad5dcee159: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 19:03:26.526115 16837 raft_consensus.cc:729] T 12e679a662f746faa893b0d9edabbb29 P 2c266e5fa9d94019907157fb874402b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c266e5fa9d94019907157fb874402b1, State: Initialized, Role: FOLLOWER
I20260929 19:03:26.526155 16837 consensus_queue.cc:260] T 12e679a662f746faa893b0d9edabbb29 P 2c266e5fa9d94019907157fb874402b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } }
I20260929 19:03:26.526242 16837 ts_tablet_manager.cc:1428] T 12e679a662f746faa893b0d9edabbb29 P 2c266e5fa9d94019907157fb874402b1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:26.526243 16836 raft_consensus.cc:348] T 12e679a662f746faa893b0d9edabbb29 P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } }
I20260929 19:03:26.526307 16836 raft_consensus.cc:374] T 12e679a662f746faa893b0d9edabbb29 P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:26.526368 16836 raft_consensus.cc:729] T 12e679a662f746faa893b0d9edabbb29 P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ea6b6f3d04e4b16aa63c8ad5dcee159, State: Initialized, Role: FOLLOWER
I20260929 19:03:26.526417 16836 consensus_queue.cc:260] T 12e679a662f746faa893b0d9edabbb29 P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } }
I20260929 19:03:26.526504 16836 ts_tablet_manager.cc:1428] T 12e679a662f746faa893b0d9edabbb29 P 9ea6b6f3d04e4b16aa63c8ad5dcee159: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260929 19:03:26.527036 16840 tablet_bootstrap.cc:492] T 12e679a662f746faa893b0d9edabbb29 P 720cc0480d0e4badabac18d7ff7e8bf4: No bootstrap required, opened a new log
I20260929 19:03:26.527078 16840 ts_tablet_manager.cc:1397] T 12e679a662f746faa893b0d9edabbb29 P 720cc0480d0e4badabac18d7ff7e8bf4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:03:26.527169 16840 raft_consensus.cc:348] T 12e679a662f746faa893b0d9edabbb29 P 720cc0480d0e4badabac18d7ff7e8bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } }
I20260929 19:03:26.527230 16840 raft_consensus.cc:374] T 12e679a662f746faa893b0d9edabbb29 P 720cc0480d0e4badabac18d7ff7e8bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:26.527249 16840 raft_consensus.cc:729] T 12e679a662f746faa893b0d9edabbb29 P 720cc0480d0e4badabac18d7ff7e8bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 720cc0480d0e4badabac18d7ff7e8bf4, State: Initialized, Role: FOLLOWER
I20260929 19:03:26.527292 16840 consensus_queue.cc:260] T 12e679a662f746faa893b0d9edabbb29 P 720cc0480d0e4badabac18d7ff7e8bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } }
I20260929 19:03:26.527369 16840 ts_tablet_manager.cc:1428] T 12e679a662f746faa893b0d9edabbb29 P 720cc0480d0e4badabac18d7ff7e8bf4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:26.721026 16844 raft_consensus.cc:482] T 12e679a662f746faa893b0d9edabbb29 P 720cc0480d0e4badabac18d7ff7e8bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:03:26.721123 16844 raft_consensus.cc:504] T 12e679a662f746faa893b0d9edabbb29 P 720cc0480d0e4badabac18d7ff7e8bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } }
I20260929 19:03:26.721304 16844 leader_election.cc:288] T 12e679a662f746faa893b0d9edabbb29 P 720cc0480d0e4badabac18d7ff7e8bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c266e5fa9d94019907157fb874402b1 (127.8.158.65:34403), 9ea6b6f3d04e4b16aa63c8ad5dcee159 (127.8.158.67:34541)
I20260929 19:03:26.721588 16485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e679a662f746faa893b0d9edabbb29" candidate_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c266e5fa9d94019907157fb874402b1" is_pre_election: true
I20260929 19:03:26.721669 16485 raft_consensus.cc:2454] T 12e679a662f746faa893b0d9edabbb29 P 2c266e5fa9d94019907157fb874402b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 720cc0480d0e4badabac18d7ff7e8bf4 in term 0.
I20260929 19:03:26.721592 16756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e679a662f746faa893b0d9edabbb29" candidate_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" is_pre_election: true
I20260929 19:03:26.721714 16756 raft_consensus.cc:2454] T 12e679a662f746faa893b0d9edabbb29 P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 720cc0480d0e4badabac18d7ff7e8bf4 in term 0.
I20260929 19:03:26.721838 16558 leader_election.cc:302] T 12e679a662f746faa893b0d9edabbb29 P 720cc0480d0e4badabac18d7ff7e8bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c266e5fa9d94019907157fb874402b1, 720cc0480d0e4badabac18d7ff7e8bf4; no voters:
I20260929 19:03:26.721966 16844 raft_consensus.cc:2781] T 12e679a662f746faa893b0d9edabbb29 P 720cc0480d0e4badabac18d7ff7e8bf4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:03:26.722018 16844 raft_consensus.cc:482] T 12e679a662f746faa893b0d9edabbb29 P 720cc0480d0e4badabac18d7ff7e8bf4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:03:26.722033 16844 raft_consensus.cc:3037] T 12e679a662f746faa893b0d9edabbb29 P 720cc0480d0e4badabac18d7ff7e8bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:26.722752 16844 raft_consensus.cc:504] T 12e679a662f746faa893b0d9edabbb29 P 720cc0480d0e4badabac18d7ff7e8bf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } }
I20260929 19:03:26.722854 16844 leader_election.cc:288] T 12e679a662f746faa893b0d9edabbb29 P 720cc0480d0e4badabac18d7ff7e8bf4 [CANDIDATE]: Term 1 election: Requested vote from peers 2c266e5fa9d94019907157fb874402b1 (127.8.158.65:34403), 9ea6b6f3d04e4b16aa63c8ad5dcee159 (127.8.158.67:34541)
I20260929 19:03:26.723070 16756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e679a662f746faa893b0d9edabbb29" candidate_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159"
I20260929 19:03:26.723104 16485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12e679a662f746faa893b0d9edabbb29" candidate_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c266e5fa9d94019907157fb874402b1"
I20260929 19:03:26.723169 16756 raft_consensus.cc:3037] T 12e679a662f746faa893b0d9edabbb29 P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:26.723184 16485 raft_consensus.cc:3037] T 12e679a662f746faa893b0d9edabbb29 P 2c266e5fa9d94019907157fb874402b1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:26.723759 16756 raft_consensus.cc:2454] T 12e679a662f746faa893b0d9edabbb29 P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 720cc0480d0e4badabac18d7ff7e8bf4 in term 1.
I20260929 19:03:26.723906 16485 raft_consensus.cc:2454] T 12e679a662f746faa893b0d9edabbb29 P 2c266e5fa9d94019907157fb874402b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 720cc0480d0e4badabac18d7ff7e8bf4 in term 1.
I20260929 19:03:26.724066 16558 leader_election.cc:302] T 12e679a662f746faa893b0d9edabbb29 P 720cc0480d0e4badabac18d7ff7e8bf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 720cc0480d0e4badabac18d7ff7e8bf4, 9ea6b6f3d04e4b16aa63c8ad5dcee159; no voters:
I20260929 19:03:26.724177 16844 raft_consensus.cc:2781] T 12e679a662f746faa893b0d9edabbb29 P 720cc0480d0e4badabac18d7ff7e8bf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:03:26.724252 16844 raft_consensus.cc:686] T 12e679a662f746faa893b0d9edabbb29 P 720cc0480d0e4badabac18d7ff7e8bf4 [term 1 LEADER]: Becoming Leader. State: Replica: 720cc0480d0e4badabac18d7ff7e8bf4, State: Running, Role: LEADER
I20260929 19:03:26.724287 16844 consensus_queue.cc:237] T 12e679a662f746faa893b0d9edabbb29 P 720cc0480d0e4badabac18d7ff7e8bf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } }
I20260929 19:03:26.724920 16326 catalog_manager.cc:5495] T 12e679a662f746faa893b0d9edabbb29 P 720cc0480d0e4badabac18d7ff7e8bf4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 720cc0480d0e4badabac18d7ff7e8bf4 (127.8.158.66). New cstate: current_term: 1 leader_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } health_report { overall_health: UNKNOWN } } }
I20260929 19:03:26.753170 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:26.750859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49536 (local address 127.8.158.66:36373)
0929 19:03:26.750921 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:26.750923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:26.750933 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:03:26.751005 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:26.751006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:26.751022 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:26.751086 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:03:26.751221 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.751692 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.752471 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.752672 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.752750 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:26.752907 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:03:26.752943 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:03:26.753004 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:26.753088 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 19:03:26.753107 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 19:03:26.753887 16756 raft_consensus.cc:1261] T 12e679a662f746faa893b0d9edabbb29 P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [term 1 FOLLOWER]: Refusing update from remote peer 720cc0480d0e4badabac18d7ff7e8bf4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:03:26.753944 16485 raft_consensus.cc:1261] T 12e679a662f746faa893b0d9edabbb29 P 2c266e5fa9d94019907157fb874402b1 [term 1 FOLLOWER]: Refusing update from remote peer 720cc0480d0e4badabac18d7ff7e8bf4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:03:26.754151 16856 consensus_queue.cc:1035] T 12e679a662f746faa893b0d9edabbb29 P 720cc0480d0e4badabac18d7ff7e8bf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:03:26.754266 16844 consensus_queue.cc:1035] T 12e679a662f746faa893b0d9edabbb29 P 720cc0480d0e4badabac18d7ff7e8bf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:03:26.756809 16328 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:03:26.756892 16328 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:03:26.757493 16328 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 61f4a227b7324cb584378135dfc9874a: Sending DeleteTablet for 3 replicas of tablet 12e679a662f746faa893b0d9edabbb29
I20260929 19:03:26.757752 16465 tablet_service.cc:1514] Processing DeleteTablet for tablet 12e679a662f746faa893b0d9edabbb29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:26 UTC) from {username='oryx', principal='oryx/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:41502
I20260929 19:03:26.757783 16605 tablet_service.cc:1514] Processing DeleteTablet for tablet 12e679a662f746faa893b0d9edabbb29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:26 UTC) from {username='oryx', principal='oryx/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:49494
I20260929 19:03:26.757848 16734 tablet_service.cc:1514] Processing DeleteTablet for tablet 12e679a662f746faa893b0d9edabbb29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:26 UTC) from {username='oryx', principal='oryx/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:34246
I20260929 19:03:26.757939 16859 tablet_replica.cc:330] T 12e679a662f746faa893b0d9edabbb29 P 9ea6b6f3d04e4b16aa63c8ad5dcee159: stopping tablet replica
I20260929 19:03:26.757941 16860 tablet_replica.cc:330] T 12e679a662f746faa893b0d9edabbb29 P 2c266e5fa9d94019907157fb874402b1: stopping tablet replica
I20260929 19:03:26.757993 16859 raft_consensus.cc:2229] T 12e679a662f746faa893b0d9edabbb29 P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:03:26.758001 16860 raft_consensus.cc:2229] T 12e679a662f746faa893b0d9edabbb29 P 2c266e5fa9d94019907157fb874402b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:03:26.758046 16859 raft_consensus.cc:2258] T 12e679a662f746faa893b0d9edabbb29 P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:03:26.758059 16860 raft_consensus.cc:2258] T 12e679a662f746faa893b0d9edabbb29 P 2c266e5fa9d94019907157fb874402b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:03:26.758124 16861 tablet_replica.cc:330] T 12e679a662f746faa893b0d9edabbb29 P 720cc0480d0e4badabac18d7ff7e8bf4: stopping tablet replica
I20260929 19:03:26.758194 16861 raft_consensus.cc:2229] T 12e679a662f746faa893b0d9edabbb29 P 720cc0480d0e4badabac18d7ff7e8bf4 [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:03:26.758255 16861 raft_consensus.cc:2258] T 12e679a662f746faa893b0d9edabbb29 P 720cc0480d0e4badabac18d7ff7e8bf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:03:26.758224 16328 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:03:26.758386 16328 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:03:26.758586 16861 ts_tablet_manager.cc:1905] T 12e679a662f746faa893b0d9edabbb29 P 720cc0480d0e4badabac18d7ff7e8bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:03:26.758661 16860 ts_tablet_manager.cc:1905] T 12e679a662f746faa893b0d9edabbb29 P 2c266e5fa9d94019907157fb874402b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:03:26.758885 16859 ts_tablet_manager.cc:1905] T 12e679a662f746faa893b0d9edabbb29 P 9ea6b6f3d04e4b16aa63c8ad5dcee159: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:03:26.760097 16860 ts_tablet_manager.cc:1918] T 12e679a662f746faa893b0d9edabbb29 P 2c266e5fa9d94019907157fb874402b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:03:26.760120 16861 ts_tablet_manager.cc:1918] T 12e679a662f746faa893b0d9edabbb29 P 720cc0480d0e4badabac18d7ff7e8bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:03:26.760180 16860 log.cc:1196] T 12e679a662f746faa893b0d9edabbb29 P 2c266e5fa9d94019907157fb874402b1: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-0/wal/wals/12e679a662f746faa893b0d9edabbb29
I20260929 19:03:26.760185 16861 log.cc:1196] T 12e679a662f746faa893b0d9edabbb29 P 720cc0480d0e4badabac18d7ff7e8bf4: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-1/wal/wals/12e679a662f746faa893b0d9edabbb29
I20260929 19:03:26.760293 16859 ts_tablet_manager.cc:1918] T 12e679a662f746faa893b0d9edabbb29 P 9ea6b6f3d04e4b16aa63c8ad5dcee159: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:03:26.760351 16859 log.cc:1196] T 12e679a662f746faa893b0d9edabbb29 P 9ea6b6f3d04e4b16aa63c8ad5dcee159: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-2/wal/wals/12e679a662f746faa893b0d9edabbb29
I20260929 19:03:26.760413 16860 ts_tablet_manager.cc:1939] T 12e679a662f746faa893b0d9edabbb29 P 2c266e5fa9d94019907157fb874402b1: Deleting consensus metadata
I20260929 19:03:26.760435 16861 ts_tablet_manager.cc:1939] T 12e679a662f746faa893b0d9edabbb29 P 720cc0480d0e4badabac18d7ff7e8bf4: Deleting consensus metadata
I20260929 19:03:26.760560 16859 ts_tablet_manager.cc:1939] T 12e679a662f746faa893b0d9edabbb29 P 9ea6b6f3d04e4b16aa63c8ad5dcee159: Deleting consensus metadata
I20260929 19:03:26.760727 16314 catalog_manager.cc:4841] TS 2c266e5fa9d94019907157fb874402b1 (127.8.158.65:34403): tablet 12e679a662f746faa893b0d9edabbb29 (table test-table [id=d886f776ae7544ad9897b502d39b66ee]) successfully deleted
I20260929 19:03:26.760828 16314 catalog_manager.cc:4841] TS 9ea6b6f3d04e4b16aa63c8ad5dcee159 (127.8.158.67:34541): tablet 12e679a662f746faa893b0d9edabbb29 (table test-table [id=d886f776ae7544ad9897b502d39b66ee]) successfully deleted
I20260929 19:03:26.760934 16312 catalog_manager.cc:4841] TS 720cc0480d0e4badabac18d7ff7e8bf4 (127.8.158.66:36373): tablet 12e679a662f746faa893b0d9edabbb29 (table test-table [id=d886f776ae7544ad9897b502d39b66ee]) successfully deleted
I20260929 19:03:26.761503 16605 tablet_service.cc:1467] Processing CreateTablet for tablet d2d164fb7bcd4e6eb6fc3a5276ed81ca (DEFAULT_TABLE table=test-table [id=bb0b2d8967a84bccbffc08a78ea2f7f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:26.761519 16465 tablet_service.cc:1467] Processing CreateTablet for tablet d2d164fb7bcd4e6eb6fc3a5276ed81ca (DEFAULT_TABLE table=test-table [id=bb0b2d8967a84bccbffc08a78ea2f7f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:26.761574 16734 tablet_service.cc:1467] Processing CreateTablet for tablet d2d164fb7bcd4e6eb6fc3a5276ed81ca (DEFAULT_TABLE table=test-table [id=bb0b2d8967a84bccbffc08a78ea2f7f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:26.761607 16465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2d164fb7bcd4e6eb6fc3a5276ed81ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:26.761606 16605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2d164fb7bcd4e6eb6fc3a5276ed81ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:26.761682 16734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2d164fb7bcd4e6eb6fc3a5276ed81ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:26.762534 16836 tablet_bootstrap.cc:492] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 9ea6b6f3d04e4b16aa63c8ad5dcee159: Bootstrap starting.
I20260929 19:03:26.762741 16837 tablet_bootstrap.cc:492] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 2c266e5fa9d94019907157fb874402b1: Bootstrap starting.
I20260929 19:03:26.763063 16837 tablet_bootstrap.cc:654] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 2c266e5fa9d94019907157fb874402b1: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:26.763178 16836 tablet_bootstrap.cc:654] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 9ea6b6f3d04e4b16aa63c8ad5dcee159: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:26.763320 16840 tablet_bootstrap.cc:492] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 720cc0480d0e4badabac18d7ff7e8bf4: Bootstrap starting.
I20260929 19:03:26.763633 16840 tablet_bootstrap.cc:654] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 720cc0480d0e4badabac18d7ff7e8bf4: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:26.763767 16837 tablet_bootstrap.cc:492] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 2c266e5fa9d94019907157fb874402b1: No bootstrap required, opened a new log
I20260929 19:03:26.763806 16837 ts_tablet_manager.cc:1397] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 2c266e5fa9d94019907157fb874402b1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:26.763911 16837 raft_consensus.cc:348] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 2c266e5fa9d94019907157fb874402b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } }
I20260929 19:03:26.763993 16837 raft_consensus.cc:374] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 2c266e5fa9d94019907157fb874402b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:26.764029 16837 raft_consensus.cc:729] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 2c266e5fa9d94019907157fb874402b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c266e5fa9d94019907157fb874402b1, State: Initialized, Role: FOLLOWER
I20260929 19:03:26.764081 16837 consensus_queue.cc:260] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 2c266e5fa9d94019907157fb874402b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } }
I20260929 19:03:26.764156 16840 tablet_bootstrap.cc:492] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 720cc0480d0e4badabac18d7ff7e8bf4: No bootstrap required, opened a new log
I20260929 19:03:26.764191 16837 ts_tablet_manager.cc:1428] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 2c266e5fa9d94019907157fb874402b1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:26.764201 16840 ts_tablet_manager.cc:1397] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 720cc0480d0e4badabac18d7ff7e8bf4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:26.764288 16840 raft_consensus.cc:348] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 720cc0480d0e4badabac18d7ff7e8bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } }
I20260929 19:03:26.764357 16840 raft_consensus.cc:374] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 720cc0480d0e4badabac18d7ff7e8bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:26.764389 16840 raft_consensus.cc:729] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 720cc0480d0e4badabac18d7ff7e8bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 720cc0480d0e4badabac18d7ff7e8bf4, State: Initialized, Role: FOLLOWER
I20260929 19:03:26.764436 16840 consensus_queue.cc:260] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 720cc0480d0e4badabac18d7ff7e8bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } }
I20260929 19:03:26.764532 16840 ts_tablet_manager.cc:1428] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 720cc0480d0e4badabac18d7ff7e8bf4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:26.764946 16836 tablet_bootstrap.cc:492] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 9ea6b6f3d04e4b16aa63c8ad5dcee159: No bootstrap required, opened a new log
I20260929 19:03:26.764998 16836 ts_tablet_manager.cc:1397] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 9ea6b6f3d04e4b16aa63c8ad5dcee159: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 19:03:26.765129 16836 raft_consensus.cc:348] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } }
I20260929 19:03:26.765182 16836 raft_consensus.cc:374] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:26.765195 16836 raft_consensus.cc:729] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ea6b6f3d04e4b16aa63c8ad5dcee159, State: Initialized, Role: FOLLOWER
I20260929 19:03:26.765246 16836 consensus_queue.cc:260] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } }
I20260929 19:03:26.765357 16836 ts_tablet_manager.cc:1428] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 9ea6b6f3d04e4b16aa63c8ad5dcee159: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:26.839557 16842 raft_consensus.cc:482] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:03:26.839689 16842 raft_consensus.cc:504] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } }
I20260929 19:03:26.840073 16842 leader_election.cc:288] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c266e5fa9d94019907157fb874402b1 (127.8.158.65:34403), 720cc0480d0e4badabac18d7ff7e8bf4 (127.8.158.66:36373)
I20260929 19:03:26.843045 16834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:26.840282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:32985 (local address 127.8.158.65:34403)
0929 19:03:26.840343 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:26.840345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:26.840415 (+ 70us) server_negotiation.cc:407] Connection header received
0929 19:03:26.840536 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:26.840537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:26.840554 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:26.840622 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:26.840790 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.841255 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.842346 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.842698 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.842784 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:26.842795 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:26.842951 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 19:03:26.842984 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 19:03:26.843112 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:26.840467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:45835 (local address 127.8.158.66:36373)
0929 19:03:26.840532 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:26.840535 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:26.840542 (+ 7us) server_negotiation.cc:407] Connection header received
0929 19:03:26.840567 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:26.840568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:26.840576 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:26.840629 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:26.840813 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.841282 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.842367 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.842731 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.842826 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:26.842836 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:26.843013 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 19:03:26.843044 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 19:03:26.843164 16917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:26.840266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.66:36373 (local address 127.8.158.67:45835)
0929 19:03:26.840411 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:26.840421 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:26.840499 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:26.840631 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:26.840632 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:26.840634 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:26.840742 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:26.840743 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.841409 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.841410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:26.842235 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:26.842237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.842838 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.842843 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:26.842918 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:26.842922 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:03:26.843004 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 19:03:26.843042 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":17,"thread_start_us":93,"threads_started":1}
I20260929 19:03:26.843071 16916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:26.840207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:34403 (local address 127.8.158.67:32985)
0929 19:03:26.840341 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:26.840353 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:26.840450 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:26.840627 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:26.840627 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:26.840629 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:26.840719 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:26.840721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.841354 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.841354 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:26.842226 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:26.842229 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.842804 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.842805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:26.842868 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:26.842871 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:03:26.842934 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 19:03:26.842973 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":3,"thread_start_us":69,"threads_started":1}
I20260929 19:03:26.843382 16625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d164fb7bcd4e6eb6fc3a5276ed81ca" candidate_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" is_pre_election: true
I20260929 19:03:26.843511 16625 raft_consensus.cc:2454] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 720cc0480d0e4badabac18d7ff7e8bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ea6b6f3d04e4b16aa63c8ad5dcee159 in term 0.
I20260929 19:03:26.843559 16485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d164fb7bcd4e6eb6fc3a5276ed81ca" candidate_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c266e5fa9d94019907157fb874402b1" is_pre_election: true
I20260929 19:03:26.843632 16485 raft_consensus.cc:2454] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 2c266e5fa9d94019907157fb874402b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ea6b6f3d04e4b16aa63c8ad5dcee159 in term 0.
I20260929 19:03:26.843675 16696 leader_election.cc:302] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 720cc0480d0e4badabac18d7ff7e8bf4, 9ea6b6f3d04e4b16aa63c8ad5dcee159; no voters:
I20260929 19:03:26.843804 16842 raft_consensus.cc:2781] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:03:26.843849 16842 raft_consensus.cc:482] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:03:26.843869 16842 raft_consensus.cc:3037] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:26.844522 16842 raft_consensus.cc:504] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } }
I20260929 19:03:26.844661 16842 leader_election.cc:288] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [CANDIDATE]: Term 1 election: Requested vote from peers 2c266e5fa9d94019907157fb874402b1 (127.8.158.65:34403), 720cc0480d0e4badabac18d7ff7e8bf4 (127.8.158.66:36373)
I20260929 19:03:26.844794 16485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d164fb7bcd4e6eb6fc3a5276ed81ca" candidate_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c266e5fa9d94019907157fb874402b1"
I20260929 19:03:26.844877 16485 raft_consensus.cc:3037] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 2c266e5fa9d94019907157fb874402b1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:26.844844 16625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d164fb7bcd4e6eb6fc3a5276ed81ca" candidate_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "720cc0480d0e4badabac18d7ff7e8bf4"
I20260929 19:03:26.844935 16625 raft_consensus.cc:3037] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 720cc0480d0e4badabac18d7ff7e8bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:26.845422 16485 raft_consensus.cc:2454] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 2c266e5fa9d94019907157fb874402b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ea6b6f3d04e4b16aa63c8ad5dcee159 in term 1.
I20260929 19:03:26.845609 16625 raft_consensus.cc:2454] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 720cc0480d0e4badabac18d7ff7e8bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ea6b6f3d04e4b16aa63c8ad5dcee159 in term 1.
I20260929 19:03:26.845618 16698 leader_election.cc:302] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c266e5fa9d94019907157fb874402b1, 9ea6b6f3d04e4b16aa63c8ad5dcee159; no voters:
I20260929 19:03:26.845718 16842 raft_consensus.cc:2781] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:03:26.845880 16842 raft_consensus.cc:686] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [term 1 LEADER]: Becoming Leader. State: Replica: 9ea6b6f3d04e4b16aa63c8ad5dcee159, State: Running, Role: LEADER
I20260929 19:03:26.845956 16842 consensus_queue.cc:237] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } }
I20260929 19:03:26.846664 16326 catalog_manager.cc:5495] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 9ea6b6f3d04e4b16aa63c8ad5dcee159 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ea6b6f3d04e4b16aa63c8ad5dcee159 (127.8.158.67). New cstate: current_term: 1 leader_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ea6b6f3d04e4b16aa63c8ad5dcee159" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 34541 } health_report { overall_health: HEALTHY } } }
I20260929 19:03:26.861819 16920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:26.859690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34250 (local address 127.8.158.67:34541)
0929 19:03:26.859816 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:26.859817 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:26.859824 (+ 7us) server_negotiation.cc:407] Connection header received
0929 19:03:26.859849 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:26.859849 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:26.859865 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:26.859933 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:03:26.860013 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.860465 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.861164 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.861344 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.861397 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:26.861523 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:03:26.861631 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:03:26.861689 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:26.861752 (+ 63us) server_negotiation.cc:299] Negotiation successful
0929 19:03:26.861767 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
I20260929 19:03:26.866530 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:26.864292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:53493 (local address 127.8.158.66:36373)
0929 19:03:26.864340 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:26.864342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:26.864351 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:03:26.864417 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:26.864418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:26.864427 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:26.864481 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:03:26.864648 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.865091 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.865839 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.866025 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.866104 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:26.866273 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:03:26.866307 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:03:26.866393 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:26.866446 (+ 53us) server_negotiation.cc:299] Negotiation successful
0929 19:03:26.866464 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260929 19:03:26.866582 16923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:26.864164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.66:36373 (local address 127.8.158.67:53493)
0929 19:03:26.864302 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:26.864313 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:26.864365 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:26.864484 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:26.864485 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:26.864486 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 19:03:26.864574 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:26.864575 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.865170 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.865171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:26.865741 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:26.865742 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.866102 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.866103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:26.866172 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:26.866183 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 19:03:26.866394 (+ 211us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 19:03:26.866396 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 19:03:26.866444 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 19:03:26.866485 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":75,"threads_started":1}
I20260929 19:03:26.868436 16485 raft_consensus.cc:1261] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 2c266e5fa9d94019907157fb874402b1 [term 1 FOLLOWER]: Refusing update from remote peer 9ea6b6f3d04e4b16aa63c8ad5dcee159: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:03:26.868450 16625 raft_consensus.cc:1261] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 720cc0480d0e4badabac18d7ff7e8bf4 [term 1 FOLLOWER]: Refusing update from remote peer 9ea6b6f3d04e4b16aa63c8ad5dcee159: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:03:26.868721 16842 consensus_queue.cc:1035] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c266e5fa9d94019907157fb874402b1" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 34403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:03:26.868817 16918 consensus_queue.cc:1035] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "720cc0480d0e4badabac18d7ff7e8bf4" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 36373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:03:26.875303 16920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:26.873089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:34907 (local address 127.8.158.67:34541)
0929 19:03:26.873124 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:26.873126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:26.873149 (+ 23us) server_negotiation.cc:407] Connection header received
0929 19:03:26.873169 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:26.873170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:26.873186 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:26.873246 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:03:26.873353 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.873841 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.874591 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.874761 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.874812 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:26.875013 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:03:26.875061 (+ 48us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:03:26.875125 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:26.875212 (+ 87us) server_negotiation.cc:299] Negotiation successful
0929 19:03:26.875235 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 19:03:26.875335 16890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:26.873029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.67:34541 (local address 127.8.158.66:34907)
0929 19:03:26.873064 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:26.873075 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:26.873120 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:26.873252 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:26.873252 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:26.873254 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 19:03:26.873290 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:26.873291 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.873908 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.873909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:26.874521 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:26.874523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:26.874870 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:26.874871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:26.874924 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:26.874930 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 19:03:26.875137 (+ 207us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 19:03:26.875139 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 19:03:26.875195 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 19:03:26.875261 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260929 19:03:26.882000 16328 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:03:26.882092 16328 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:03:26.882776 16328 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 61f4a227b7324cb584378135dfc9874a: Sending DeleteTablet for 3 replicas of tablet d2d164fb7bcd4e6eb6fc3a5276ed81ca
I20260929 19:03:26.883014 16734 tablet_service.cc:1514] Processing DeleteTablet for tablet d2d164fb7bcd4e6eb6fc3a5276ed81ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:26 UTC) from {username='oryx', principal='oryx/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:34246
I20260929 19:03:26.883023 16605 tablet_service.cc:1514] Processing DeleteTablet for tablet d2d164fb7bcd4e6eb6fc3a5276ed81ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:26 UTC) from {username='oryx', principal='oryx/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:49494
I20260929 19:03:26.883107 16859 tablet_replica.cc:330] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 9ea6b6f3d04e4b16aa63c8ad5dcee159: stopping tablet replica
I20260929 19:03:26.883162 16859 raft_consensus.cc:2229] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:03:26.883172 16861 tablet_replica.cc:330] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 720cc0480d0e4badabac18d7ff7e8bf4: stopping tablet replica
I20260929 19:03:26.883222 16861 raft_consensus.cc:2229] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 720cc0480d0e4badabac18d7ff7e8bf4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:03:26.883234 16859 raft_consensus.cc:2258] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 9ea6b6f3d04e4b16aa63c8ad5dcee159 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:03:26.883263 16861 raft_consensus.cc:2258] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 720cc0480d0e4badabac18d7ff7e8bf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:03:26.883504 16859 ts_tablet_manager.cc:1905] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 9ea6b6f3d04e4b16aa63c8ad5dcee159: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:03:26.883608 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 16400
I20260929 19:03:26.885237 16859 ts_tablet_manager.cc:1918] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 9ea6b6f3d04e4b16aa63c8ad5dcee159: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 19:03:26.885298 16859 log.cc:1196] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 9ea6b6f3d04e4b16aa63c8ad5dcee159: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-2/wal/wals/d2d164fb7bcd4e6eb6fc3a5276ed81ca
I20260929 19:03:26.885505 16859 ts_tablet_manager.cc:1939] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 9ea6b6f3d04e4b16aa63c8ad5dcee159: Deleting consensus metadata
I20260929 19:03:26.886471 16861 ts_tablet_manager.cc:1905] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 720cc0480d0e4badabac18d7ff7e8bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:03:26.888202 16861 ts_tablet_manager.cc:1918] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 720cc0480d0e4badabac18d7ff7e8bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 19:03:26.888252 16861 log.cc:1196] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 720cc0480d0e4badabac18d7ff7e8bf4: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708573736807-8825-0/minicluster-data/ts-1/wal/wals/d2d164fb7bcd4e6eb6fc3a5276ed81ca
I20260929 19:03:26.888450 16861 ts_tablet_manager.cc:1939] T d2d164fb7bcd4e6eb6fc3a5276ed81ca P 720cc0480d0e4badabac18d7ff7e8bf4: Deleting consensus metadata
I20260929 19:03:26.888774 16312 catalog_manager.cc:4841] TS 720cc0480d0e4badabac18d7ff7e8bf4 (127.8.158.66:36373): tablet d2d164fb7bcd4e6eb6fc3a5276ed81ca (table test-table [id=bb0b2d8967a84bccbffc08a78ea2f7f3]) successfully deleted
I20260929 19:03:26.893198 16314 catalog_manager.cc:4841] TS 9ea6b6f3d04e4b16aa63c8ad5dcee159 (127.8.158.67:34541): tablet d2d164fb7bcd4e6eb6fc3a5276ed81ca (table test-table [id=bb0b2d8967a84bccbffc08a78ea2f7f3]) successfully deleted
W20260929 19:03:26.893414 16314 connection.cc:535] client connection to 127.8.158.65:34403 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260929 19:03:26.893501 16314 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 19:03:26.893590 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 16540
I20260929 19:03:26.898501 16929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:26.893822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:34403 (local address 127.0.0.1:41526)
0929 19:03:26.898414 (+ 4592us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:26.898461 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.65:34403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4575,"thread_start_us":38,"threads_started":1}
W20260929 19:03:26.898597 16314 catalog_manager.cc:4584] TS 2c266e5fa9d94019907157fb874402b1 (127.8.158.65:34403): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d2d164fb7bcd4e6eb6fc3a5276ed81ca: Network error: Client connection negotiation failed: client connection to 127.8.158.65:34403: connect: Connection refused (error 111)
I20260929 19:03:26.900164 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 16680
I20260929 19:03:26.905164 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 16296
2026-09-29T19:03:26Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2787 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708573736807-8825-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:03:26 dist-test-slave-nmsz krb5kdc[16932](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:03:26 dist-test-slave-nmsz krb5kdc[16932](info): set up 2 sockets
Sep 29 19:03:26 dist-test-slave-nmsz krb5kdc[16932](info): commencing operation
krb5kdc: starting...
W20260929 19:03:27.962875 8825 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Sep 29 19:03:28 dist-test-slave-nmsz krb5kdc[16932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:03:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:03:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.158.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
I20260929 19:03:28.080906 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:33601
--webserver_interface=127.8.158.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:34057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:33601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:28.160079 16945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:28.160271 16945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:28.160293 16945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:28.161433 16945 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:03:28.161471 16945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:28.161487 16945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:28.161501 16945 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:03:28.161537 16945 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:03:28.163048 16945 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:34057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:33601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:33601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.158.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:28.163285 16945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:28.163509 16945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:28.168259 16950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:28.168313 16953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:28.168313 16951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:28.168507 16945 server_base.cc:1034] running on GCE node
I20260929 19:03:28.168727 16945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:28.168977 16945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:28.170163 16945 hybrid_clock.cc:648] HybridClock initialized: now 1790708608170147 us; error 28 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T19:03:28Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1272 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:03:28 dist-test-slave-nmsz krb5kdc[16958](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:03:28 dist-test-slave-nmsz krb5kdc[16958](info): set up 2 sockets
Sep 29 19:03:28 dist-test-slave-nmsz krb5kdc[16958](info): commencing operation
krb5kdc: starting...
W20260929 19:03:29.259315 8825 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Sep 29 19:03:29 dist-test-slave-nmsz krb5kdc[16958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:03:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:03:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.158.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
I20260929 19:03:29.376847 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:40017
--webserver_interface=127.8.158.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:38173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:40017
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:29.456189 16971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:29.456429 16971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:29.456465 16971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:29.457584 16971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:03:29.457653 16971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:29.457679 16971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:29.457705 16971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:03:29.457736 16971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:03:29.459280 16971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:38173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:40017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:40017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.158.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:29.459553 16971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:29.459780 16971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:29.464609 16979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:29.464654 16976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:29.464612 16977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:29.464835 16971 server_base.cc:1034] running on GCE node
I20260929 19:03:29.465144 16971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:29.465382 16971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:29.466549 16971 hybrid_clock.cc:648] HybridClock initialized: now 1790708609466537 us; error 30 us; skew 500 ppm
Sep 29 19:03:29 dist-test-slave-nmsz krb5kdc[16958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708609, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:29.469129 16971 init.cc:377] Logged in from keytab as kudu/127.8.158.126@KRBTEST.COM (short username kudu)
I20260929 19:03:29.469612 16971 webserver.cc:460] Webserver started at http://127.8.158.126:45977/ using document root <none> and password file <none>
I20260929 19:03:29.469847 16971 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:29.469914 16971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:29.470068 16971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:29.471156 16971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "0a97b7cb8c8e44c088ff3ec9c6dbdeb0"
format_stamp: "Formatted at 2026-09-29 19:03:29 on dist-test-slave-nmsz"
server_key: "51b360f835fb514b48893f57e9284930"
server_key_iv: "bc82cdec1982955a67a3320c947167d9"
server_key_version: "encryptionkey@0"
I20260929 19:03:29.471519 16971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "0a97b7cb8c8e44c088ff3ec9c6dbdeb0"
format_stamp: "Formatted at 2026-09-29 19:03:29 on dist-test-slave-nmsz"
server_key: "51b360f835fb514b48893f57e9284930"
server_key_iv: "bc82cdec1982955a67a3320c947167d9"
server_key_version: "encryptionkey@0"
I20260929 19:03:29.472787 16971 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 19:03:29.473449 16986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:29.473613 16971 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:03:29.473697 16971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "0a97b7cb8c8e44c088ff3ec9c6dbdeb0"
format_stamp: "Formatted at 2026-09-29 19:03:29 on dist-test-slave-nmsz"
server_key: "51b360f835fb514b48893f57e9284930"
server_key_iv: "bc82cdec1982955a67a3320c947167d9"
server_key_version: "encryptionkey@0"
I20260929 19:03:29.473783 16971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:29.492976 16971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:29.495394 16971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:29.495572 16971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:29.499797 16971 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.126:40017
I20260929 19:03:29.499825 17038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.126:40017 every 8 connection(s)
I20260929 19:03:29.500181 16971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:03:29.500972 17039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:29.501412 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 16971
I20260929 19:03:29.501514 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:03:29.501696 8825 external_mini_cluster.cc:1428] Setting key 7b994ad21fd17b6162a3157dc302631a
I20260929 19:03:29.504432 17039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a97b7cb8c8e44c088ff3ec9c6dbdeb0: Bootstrap starting.
I20260929 19:03:29.505081 17039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a97b7cb8c8e44c088ff3ec9c6dbdeb0: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:29.505357 17039 log.cc:824] T 00000000000000000000000000000000 P 0a97b7cb8c8e44c088ff3ec9c6dbdeb0: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:29.506076 17039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a97b7cb8c8e44c088ff3ec9c6dbdeb0: No bootstrap required, opened a new log
I20260929 19:03:29.508034 17039 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a97b7cb8c8e44c088ff3ec9c6dbdeb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a97b7cb8c8e44c088ff3ec9c6dbdeb0" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 40017 } }
I20260929 19:03:29.508203 17039 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a97b7cb8c8e44c088ff3ec9c6dbdeb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:29.508236 17039 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a97b7cb8c8e44c088ff3ec9c6dbdeb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a97b7cb8c8e44c088ff3ec9c6dbdeb0, State: Initialized, Role: FOLLOWER
I20260929 19:03:29.508358 17039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a97b7cb8c8e44c088ff3ec9c6dbdeb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a97b7cb8c8e44c088ff3ec9c6dbdeb0" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 40017 } }
I20260929 19:03:29.508440 17039 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a97b7cb8c8e44c088ff3ec9c6dbdeb0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:03:29.508471 17039 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a97b7cb8c8e44c088ff3ec9c6dbdeb0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:03:29.508515 17039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a97b7cb8c8e44c088ff3ec9c6dbdeb0 [term 0 FOLLOWER]: Advancing to term 1
Sep 29 19:03:29 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:03:29.509069 17039 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a97b7cb8c8e44c088ff3ec9c6dbdeb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a97b7cb8c8e44c088ff3ec9c6dbdeb0" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 40017 } }
I20260929 19:03:29.509176 17039 leader_election.cc:302] T 00000000000000000000000000000000 P 0a97b7cb8c8e44c088ff3ec9c6dbdeb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a97b7cb8c8e44c088ff3ec9c6dbdeb0; no voters:
I20260929 19:03:29.509367 17039 leader_election.cc:288] T 00000000000000000000000000000000 P 0a97b7cb8c8e44c088ff3ec9c6dbdeb0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:03:29.509449 17044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a97b7cb8c8e44c088ff3ec9c6dbdeb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:03:29.509635 17039 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a97b7cb8c8e44c088ff3ec9c6dbdeb0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:03:29.509678 17044 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a97b7cb8c8e44c088ff3ec9c6dbdeb0 [term 1 LEADER]: Becoming Leader. State: Replica: 0a97b7cb8c8e44c088ff3ec9c6dbdeb0, State: Running, Role: LEADER
I20260929 19:03:29.509855 17044 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a97b7cb8c8e44c088ff3ec9c6dbdeb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a97b7cb8c8e44c088ff3ec9c6dbdeb0" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 40017 } }
I20260929 19:03:29.510277 17045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a97b7cb8c8e44c088ff3ec9c6dbdeb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a97b7cb8c8e44c088ff3ec9c6dbdeb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a97b7cb8c8e44c088ff3ec9c6dbdeb0" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 40017 } } }
I20260929 19:03:29.510557 17045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a97b7cb8c8e44c088ff3ec9c6dbdeb0 [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:29.510478 17046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a97b7cb8c8e44c088ff3ec9c6dbdeb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a97b7cb8c8e44c088ff3ec9c6dbdeb0. Latest consensus state: current_term: 1 leader_uuid: "0a97b7cb8c8e44c088ff3ec9c6dbdeb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a97b7cb8c8e44c088ff3ec9c6dbdeb0" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 40017 } } }
I20260929 19:03:29.510825 17046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a97b7cb8c8e44c088ff3ec9c6dbdeb0 [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:29.511001 17055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:03:29.511978 17055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:03:29.512871 17042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:29.502155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52610 (local address 127.8.158.126:40017)
0929 19:03:29.502330 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:29.502338 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:29.502369 (+ 31us) server_negotiation.cc:407] Connection header received
0929 19:03:29.502824 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:29.502832 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:29.502891 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:29.503036 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:29.503608 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:29.504913 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:29.505805 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:29.506416 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:29.506521 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:29.509078 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:29.509082 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:29.509083 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:29.509094 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:29.511305 (+ 2211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:29.511739 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:29.511741 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:29.511741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:29.511816 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:29.512044 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:29.512045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:29.512045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:29.512281 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:29.512383 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:29.512557 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 19:03:29.512648 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":62,"threads_started":1}
I20260929 19:03:29.513891 17055 catalog_manager.cc:1347] Generated new cluster ID: 1d30043f004f4783a53ae82a3442b67b
I20260929 19:03:29.513933 17055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:03:29.523922 17055 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:03:29.524693 17055 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:03:29.533011 17055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a97b7cb8c8e44c088ff3ec9c6dbdeb0: Generated new TSK 0
I20260929 19:03:29.533181 17055 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.158.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab.
Entry for principal kudu/127.8.158.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab.
I20260929 19:03:29.565203 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.65:0
--local_ip_for_outbound_sockets=127.8.158.65
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:40017
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab
--principal=kudu/127.8.158.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:38173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:29.646423 17065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:29.646602 17065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:29.646624 17065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:29.647706 17065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:29.647738 17065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:29.647763 17065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.65
I20260929 19:03:29.649534 17065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:38173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab
--principal=kudu/127.8.158.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:40017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.65
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:29.649698 17065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:29.649856 17065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:29.654616 17070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:29.654611 17071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:29.654625 17073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:29.654924 17065 server_base.cc:1034] running on GCE node
I20260929 19:03:29.655141 17065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:29.655335 17065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:29.656505 17065 hybrid_clock.cc:648] HybridClock initialized: now 1790708609656494 us; error 25 us; skew 500 ppm
Sep 29 19:03:29 dist-test-slave-nmsz krb5kdc[16958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708609, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:29.659163 17065 init.cc:377] Logged in from keytab as kudu/127.8.158.65@KRBTEST.COM (short username kudu)
I20260929 19:03:29.659652 17065 webserver.cc:460] Webserver started at http://127.8.158.65:41691/ using document root <none> and password file <none>
I20260929 19:03:29.659839 17065 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:29.659881 17065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:29.659987 17065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:29.661087 17065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-0/data/instance:
uuid: "511bf54a5d474c0fa73d560c2a541022"
format_stamp: "Formatted at 2026-09-29 19:03:29 on dist-test-slave-nmsz"
server_key: "e2cfeada0552229d887b0955c674cb0e"
server_key_iv: "11644758c2fca8e99bba04366546ae4d"
server_key_version: "encryptionkey@0"
I20260929 19:03:29.661404 17065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance:
uuid: "511bf54a5d474c0fa73d560c2a541022"
format_stamp: "Formatted at 2026-09-29 19:03:29 on dist-test-slave-nmsz"
server_key: "e2cfeada0552229d887b0955c674cb0e"
server_key_iv: "11644758c2fca8e99bba04366546ae4d"
server_key_version: "encryptionkey@0"
I20260929 19:03:29.662801 17065 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:03:29.663582 17080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:29.663767 17065 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 19:03:29.663872 17065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-0/wal
uuid: "511bf54a5d474c0fa73d560c2a541022"
format_stamp: "Formatted at 2026-09-29 19:03:29 on dist-test-slave-nmsz"
server_key: "e2cfeada0552229d887b0955c674cb0e"
server_key_iv: "11644758c2fca8e99bba04366546ae4d"
server_key_version: "encryptionkey@0"
I20260929 19:03:29.663959 17065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:29.691208 17065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:29.693334 17065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:29.693512 17065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:29.693769 17065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:29.694084 17065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:29.694156 17065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:29.694202 17065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:29.694233 17065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:29.700314 17065 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.65:38095
I20260929 19:03:29.700356 17193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.65:38095 every 8 connection(s)
I20260929 19:03:29.700728 17065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
Sep 29 19:03:29 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708609, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:03:29.710089 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 17065
I20260929 19:03:29.710208 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance
I20260929 19:03:29.710479 8825 external_mini_cluster.cc:1428] Setting key c8e5c0f02f7808b7a251237fec5ee124
I20260929 19:03:29.713538 17042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:29.701502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:49617 (local address 127.8.158.126:40017)
0929 19:03:29.701573 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:29.701575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:29.701732 (+ 157us) server_negotiation.cc:407] Connection header received
0929 19:03:29.702182 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:29.702183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:29.702192 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:29.702251 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:29.703147 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:29.703760 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:29.705576 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:29.705957 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:29.706002 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:29.708599 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:29.708601 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:29.708601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:29.708608 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:29.711684 (+ 3076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:29.712309 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:29.712310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:29.712310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:29.712385 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:29.712757 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:29.712759 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:29.712760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:29.712919 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:29.712964 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:29.713326 (+ 362us) server_negotiation.cc:299] Negotiation successful
0929 19:03:29.713420 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 19:03:29.713845 17196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:29.701436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.65:49617)
0929 19:03:29.701608 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:29.701648 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:29.702103 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:29.702300 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:29.702303 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:29.702876 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:29.703055 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:29.703059 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:29.703887 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:29.703888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:29.705428 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:29.705431 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:29.706104 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:29.706105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:29.706232 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:29.706852 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:29.706866 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:29.708480 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:29.711782 (+ 3302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:29.711789 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:29.711792 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:29.712213 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:29.712463 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:29.712464 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:29.712464 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:29.712637 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:29.713012 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:29.713018 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:29.713156 (+ 138us) client_negotiation.cc:769] Sending connection context
0929 19:03:29.713303 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 19:03:29.713525 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":100,"threads_started":1}
I20260929 19:03:29.714359 17194 heartbeater.cc:347] Connected to a master server at 127.8.158.126:40017
I20260929 19:03:29.714452 17194 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:29.714622 17194 heartbeater.cc:511] Master 127.8.158.126:40017 requested a full tablet report, sending...
I20260929 19:03:29.715181 17003 ts_manager.cc:194] Registered new tserver with Master: 511bf54a5d474c0fa73d560c2a541022 (127.8.158.65:38095)
I20260929 19:03:29.716223 17003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.65@KRBTEST.COM'} at 127.8.158.65:49617
WARNING: no policy specified for kudu/127.8.158.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab.
Entry for principal kudu/127.8.158.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab.
I20260929 19:03:29.736979 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.66:0
--local_ip_for_outbound_sockets=127.8.158.66
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:40017
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab
--principal=kudu/127.8.158.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:38173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:29.821480 17199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:29.821674 17199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:29.821697 17199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:29.822889 17199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:29.822928 17199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:29.822955 17199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.66
I20260929 19:03:29.824347 17199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:38173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab
--principal=kudu/127.8.158.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:40017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.66
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:29.824558 17199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:29.824783 17199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:29.829665 17207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:29.829723 17205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:29.829676 17204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:29.830148 17199 server_base.cc:1034] running on GCE node
I20260929 19:03:29.830387 17199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:29.830631 17199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:29.831763 17199 hybrid_clock.cc:648] HybridClock initialized: now 1790708609831753 us; error 33 us; skew 500 ppm
Sep 29 19:03:29 dist-test-slave-nmsz krb5kdc[16958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708609, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:29.834290 17199 init.cc:377] Logged in from keytab as kudu/127.8.158.66@KRBTEST.COM (short username kudu)
I20260929 19:03:29.834713 17199 webserver.cc:460] Webserver started at http://127.8.158.66:40915/ using document root <none> and password file <none>
I20260929 19:03:29.834954 17199 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:29.835018 17199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:29.835155 17199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:29.836216 17199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-1/data/instance:
uuid: "b1bd146f80ab4630a2001e79fb795db1"
format_stamp: "Formatted at 2026-09-29 19:03:29 on dist-test-slave-nmsz"
server_key: "9147ea1d36142929c6a08e236542d7fc"
server_key_iv: "6eff3c96e039e231d299f1edeee86536"
server_key_version: "encryptionkey@0"
I20260929 19:03:29.836570 17199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance:
uuid: "b1bd146f80ab4630a2001e79fb795db1"
format_stamp: "Formatted at 2026-09-29 19:03:29 on dist-test-slave-nmsz"
server_key: "9147ea1d36142929c6a08e236542d7fc"
server_key_iv: "6eff3c96e039e231d299f1edeee86536"
server_key_version: "encryptionkey@0"
I20260929 19:03:29.837888 17199 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 19:03:29.838730 17214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:29.838920 17199 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 19:03:29.839012 17199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-1/wal
uuid: "b1bd146f80ab4630a2001e79fb795db1"
format_stamp: "Formatted at 2026-09-29 19:03:29 on dist-test-slave-nmsz"
server_key: "9147ea1d36142929c6a08e236542d7fc"
server_key_iv: "6eff3c96e039e231d299f1edeee86536"
server_key_version: "encryptionkey@0"
I20260929 19:03:29.839090 17199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:29.856751 17199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:29.858695 17199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:29.858857 17199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:29.859077 17199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:29.859386 17199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:29.859442 17199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:29.859477 17199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:29.859503 17199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:29.865302 17199 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.66:39287
I20260929 19:03:29.865347 17327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.66:39287 every 8 connection(s)
I20260929 19:03:29.865655 17199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
Sep 29 19:03:29 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708609, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.66@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:03:29.872035 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 17199
I20260929 19:03:29.872136 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance
I20260929 19:03:29.872355 8825 external_mini_cluster.cc:1428] Setting key bb6dc0371c3e0303ec8aa4094f68fdd6
I20260929 19:03:29.875293 17042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:29.866348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:51627 (local address 127.8.158.126:40017)
0929 19:03:29.866394 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:29.866397 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:29.866654 (+ 257us) server_negotiation.cc:407] Connection header received
0929 19:03:29.867266 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:29.867267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:29.867281 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:29.867334 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:29.868323 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:29.868827 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:29.869816 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:29.870023 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:29.870108 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:29.872305 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:29.872307 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:29.872307 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:29.872314 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:29.873812 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:29.874302 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:29.874303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:29.874304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:29.874398 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:29.874658 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:29.874659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:29.874659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:29.874772 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:29.874848 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:29.875129 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 19:03:29.875207 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 19:03:29.875490 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:29.866301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.66:51627)
0929 19:03:29.866499 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:29.866562 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:29.867150 (+ 588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:29.867397 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:29.867400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:29.868045 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:29.868230 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:29.868234 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:29.868923 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:29.868924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:29.869716 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:29.869718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:29.870118 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:29.870119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:29.870242 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:29.870741 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:29.870749 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:29.872206 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:29.873896 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:29.873902 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:29.873904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:29.874228 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:29.874464 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:29.874466 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:29.874466 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:29.874578 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:29.874840 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:29.874846 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:29.874988 (+ 142us) client_negotiation.cc:769] Sending connection context
0929 19:03:29.875094 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 19:03:29.875247 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":90,"threads_started":1}
I20260929 19:03:29.875883 17328 heartbeater.cc:347] Connected to a master server at 127.8.158.126:40017
I20260929 19:03:29.875972 17328 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:29.876155 17328 heartbeater.cc:511] Master 127.8.158.126:40017 requested a full tablet report, sending...
I20260929 19:03:29.876526 17003 ts_manager.cc:194] Registered new tserver with Master: b1bd146f80ab4630a2001e79fb795db1 (127.8.158.66:39287)
I20260929 19:03:29.876996 17003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.66@KRBTEST.COM'} at 127.8.158.66:51627
WARNING: no policy specified for kudu/127.8.158.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab.
Entry for principal kudu/127.8.158.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab.
I20260929 19:03:29.897809 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.67:0
--local_ip_for_outbound_sockets=127.8.158.67
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:40017
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab
--principal=kudu/127.8.158.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:38173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:29.978410 17333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:29.978601 17333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:29.978623 17333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:29.979753 17333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:29.979794 17333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:29.979821 17333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.67
I20260929 19:03:29.981235 17333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:38173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab
--principal=kudu/127.8.158.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:40017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.67
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:29.981469 17333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:29.981681 17333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:29.986696 17339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:29.986693 17341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:29.986688 17338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:29.986723 17333 server_base.cc:1034] running on GCE node
I20260929 19:03:29.987118 17333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:29.987325 17333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:29.988472 17333 hybrid_clock.cc:648] HybridClock initialized: now 1790708609988463 us; error 31 us; skew 500 ppm
Sep 29 19:03:29 dist-test-slave-nmsz krb5kdc[16958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708609, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:29.991134 17333 init.cc:377] Logged in from keytab as kudu/127.8.158.67@KRBTEST.COM (short username kudu)
I20260929 19:03:29.991526 17333 webserver.cc:460] Webserver started at http://127.8.158.67:40603/ using document root <none> and password file <none>
I20260929 19:03:29.991762 17333 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:29.991835 17333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:29.991993 17333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:29.993083 17333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-2/data/instance:
uuid: "7e17cac163394108be87d5e2932e97bc"
format_stamp: "Formatted at 2026-09-29 19:03:29 on dist-test-slave-nmsz"
server_key: "4d833949c15b2f682144dcc276c04f22"
server_key_iv: "2637ae36d63e55d6d314ded0bda2b4ef"
server_key_version: "encryptionkey@0"
I20260929 19:03:29.993458 17333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance:
uuid: "7e17cac163394108be87d5e2932e97bc"
format_stamp: "Formatted at 2026-09-29 19:03:29 on dist-test-slave-nmsz"
server_key: "4d833949c15b2f682144dcc276c04f22"
server_key_iv: "2637ae36d63e55d6d314ded0bda2b4ef"
server_key_version: "encryptionkey@0"
I20260929 19:03:29.994921 17333 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:03:29.995728 17348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:29.995905 17333 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:03:29.996031 17333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-2/wal
uuid: "7e17cac163394108be87d5e2932e97bc"
format_stamp: "Formatted at 2026-09-29 19:03:29 on dist-test-slave-nmsz"
server_key: "4d833949c15b2f682144dcc276c04f22"
server_key_iv: "2637ae36d63e55d6d314ded0bda2b4ef"
server_key_version: "encryptionkey@0"
I20260929 19:03:29.996112 17333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:30.028455 17333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:30.030421 17333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:30.030563 17333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:30.030779 17333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:30.031085 17333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:30.031137 17333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:30.031177 17333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:30.031204 17333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:30.036872 17333 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.67:46129
I20260929 19:03:30.036911 17461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.67:46129 every 8 connection(s)
I20260929 19:03:30.037236 17333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
I20260929 19:03:30.042733 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 17333
I20260929 19:03:30.042820 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance
I20260929 19:03:30.043030 8825 external_mini_cluster.cc:1428] Setting key 67a91363eb7105420b6ef6e85cea6508
Sep 29 19:03:30 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708609, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.67@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:03:30.046453 17042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:30.037911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:60613 (local address 127.8.158.126:40017)
0929 19:03:30.037965 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:30.037967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:30.038147 (+ 180us) server_negotiation.cc:407] Connection header received
0929 19:03:30.038702 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:30.038704 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:30.038716 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:30.038779 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:30.039682 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:30.040173 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:30.041122 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:30.041332 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:30.041417 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:30.043670 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:30.043671 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:30.043672 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:30.043679 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:30.044927 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:30.045405 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:30.045406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:30.045407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:30.045471 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:30.045743 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:30.045744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:30.045745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:30.045869 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:30.045955 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:30.046227 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 19:03:30.046346 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 19:03:30.046658 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:30.037902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.67:60613)
0929 19:03:30.038041 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:30.038072 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:30.038603 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:30.038854 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:30.038858 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:30.039401 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:30.039581 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:30.039585 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:30.040273 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:30.040274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:30.041034 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:30.041036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:30.041425 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:30.041426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:30.041618 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:30.042035 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:30.042043 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:30.043578 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:30.045052 (+ 1474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:30.045059 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:30.045061 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:30.045298 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:30.045551 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:30.045552 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:30.045552 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:30.045659 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:30.045945 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:30.045951 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:30.046089 (+ 138us) client_negotiation.cc:769] Sending connection context
0929 19:03:30.046204 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 19:03:30.046398 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":75,"threads_started":1}
I20260929 19:03:30.047092 17462 heartbeater.cc:347] Connected to a master server at 127.8.158.126:40017
I20260929 19:03:30.047180 17462 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:30.047351 17462 heartbeater.cc:511] Master 127.8.158.126:40017 requested a full tablet report, sending...
I20260929 19:03:30.047724 17003 ts_manager.cc:194] Registered new tserver with Master: 7e17cac163394108be87d5e2932e97bc (127.8.158.67:46129)
I20260929 19:03:30.048091 17003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.67@KRBTEST.COM'} at 127.8.158.67:60613
I20260929 19:03:30.054458 8825 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.8.158.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.158.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.158.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.65.keytab.
Entry for principal oryx/127.8.158.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.65.keytab.
WARNING: no policy specified for oryx/127.8.158.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.158.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.158.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.66.keytab.
Entry for principal oryx/127.8.158.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.66.keytab.
WARNING: no policy specified for oryx/127.8.158.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.158.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.158.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.67.keytab.
Entry for principal oryx/127.8.158.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.67.keytab.
I20260929 19:03:30.125910 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 17065
I20260929 19:03:30.130281 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 17199
I20260929 19:03:30.134366 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 17333
I20260929 19:03:30.138418 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 16971
I20260929 19:03:30.143831 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:40017
--webserver_interface=127.8.158.126
--webserver_port=45977
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:38173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:40017
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:30.225075 17471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:30.225315 17471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:30.225350 17471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:30.226622 17471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:03:30.226684 17471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:30.226711 17471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:30.226743 17471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:03:30.226769 17471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:03:30.228176 17471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:38173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:40017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:40017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.158.126
--webserver_port=45977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:30.228441 17471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:30.228658 17471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:30.233280 17477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:30.233326 17479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:30.233287 17476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:30.233600 17471 server_base.cc:1034] running on GCE node
I20260929 19:03:30.233809 17471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:30.234030 17471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:30.235175 17471 hybrid_clock.cc:648] HybridClock initialized: now 1790708610235163 us; error 28 us; skew 500 ppm
Sep 29 19:03:30 dist-test-slave-nmsz krb5kdc[16958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708610, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:30.237938 17471 init.cc:377] Logged in from keytab as kudu/127.8.158.126@KRBTEST.COM (short username kudu)
I20260929 19:03:30.238427 17471 webserver.cc:460] Webserver started at http://127.8.158.126:45977/ using document root <none> and password file <none>
I20260929 19:03:30.238653 17471 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:30.238706 17471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:30.240072 17471 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:03:30.240746 17486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:30.240923 17471 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:03:30.241020 17471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "0a97b7cb8c8e44c088ff3ec9c6dbdeb0"
format_stamp: "Formatted at 2026-09-29 19:03:29 on dist-test-slave-nmsz"
server_key: "51b360f835fb514b48893f57e9284930"
server_key_iv: "bc82cdec1982955a67a3320c947167d9"
server_key_version: "encryptionkey@0"
I20260929 19:03:30.241348 17471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:30.259899 17471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:30.262446 17471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:30.262629 17471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:30.266716 17471 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.126:40017
I20260929 19:03:30.267001 17538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.126:40017 every 8 connection(s)
I20260929 19:03:30.267253 17471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:03:30.268793 17539 sys_catalog.cc:263] Verifying existing consensus state
I20260929 19:03:30.268947 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 17471
I20260929 19:03:30.269225 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.65:38095
--local_ip_for_outbound_sockets=127.8.158.65
--tserver_master_addrs=127.8.158.126:40017
--webserver_port=41691
--webserver_interface=127.8.158.65
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab
--principal=kudu/127.8.158.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:38173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.158.65
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:03:30.269797 17539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a97b7cb8c8e44c088ff3ec9c6dbdeb0: Bootstrap starting.
I20260929 19:03:30.277750 17539 log.cc:824] T 00000000000000000000000000000000 P 0a97b7cb8c8e44c088ff3ec9c6dbdeb0: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:30.279929 17539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a97b7cb8c8e44c088ff3ec9c6dbdeb0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 19:03:30.280244 17539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a97b7cb8c8e44c088ff3ec9c6dbdeb0: Bootstrap complete.
I20260929 19:03:30.282764 17539 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a97b7cb8c8e44c088ff3ec9c6dbdeb0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a97b7cb8c8e44c088ff3ec9c6dbdeb0" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 40017 } }
I20260929 19:03:30.282986 17539 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a97b7cb8c8e44c088ff3ec9c6dbdeb0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a97b7cb8c8e44c088ff3ec9c6dbdeb0, State: Initialized, Role: FOLLOWER
I20260929 19:03:30.283154 17539 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a97b7cb8c8e44c088ff3ec9c6dbdeb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a97b7cb8c8e44c088ff3ec9c6dbdeb0" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 40017 } }
I20260929 19:03:30.283226 17539 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a97b7cb8c8e44c088ff3ec9c6dbdeb0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:03:30.283273 17539 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a97b7cb8c8e44c088ff3ec9c6dbdeb0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:03:30.283319 17539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a97b7cb8c8e44c088ff3ec9c6dbdeb0 [term 1 FOLLOWER]: Advancing to term 2
I20260929 19:03:30.284247 17539 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a97b7cb8c8e44c088ff3ec9c6dbdeb0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a97b7cb8c8e44c088ff3ec9c6dbdeb0" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 40017 } }
I20260929 19:03:30.284327 17539 leader_election.cc:302] T 00000000000000000000000000000000 P 0a97b7cb8c8e44c088ff3ec9c6dbdeb0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a97b7cb8c8e44c088ff3ec9c6dbdeb0; no voters:
I20260929 19:03:30.284649 17539 leader_election.cc:288] T 00000000000000000000000000000000 P 0a97b7cb8c8e44c088ff3ec9c6dbdeb0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 19:03:30.284682 17543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a97b7cb8c8e44c088ff3ec9c6dbdeb0 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 19:03:30.284813 17543 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a97b7cb8c8e44c088ff3ec9c6dbdeb0 [term 2 LEADER]: Becoming Leader. State: Replica: 0a97b7cb8c8e44c088ff3ec9c6dbdeb0, State: Running, Role: LEADER
I20260929 19:03:30.284953 17539 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a97b7cb8c8e44c088ff3ec9c6dbdeb0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:03:30.285012 17543 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a97b7cb8c8e44c088ff3ec9c6dbdeb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a97b7cb8c8e44c088ff3ec9c6dbdeb0" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 40017 } }
I20260929 19:03:30.285377 17545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a97b7cb8c8e44c088ff3ec9c6dbdeb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0a97b7cb8c8e44c088ff3ec9c6dbdeb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a97b7cb8c8e44c088ff3ec9c6dbdeb0" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 40017 } } }
I20260929 19:03:30.285398 17544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a97b7cb8c8e44c088ff3ec9c6dbdeb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a97b7cb8c8e44c088ff3ec9c6dbdeb0. Latest consensus state: current_term: 2 leader_uuid: "0a97b7cb8c8e44c088ff3ec9c6dbdeb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a97b7cb8c8e44c088ff3ec9c6dbdeb0" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 40017 } } }
I20260929 19:03:30.285498 17545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a97b7cb8c8e44c088ff3ec9c6dbdeb0 [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:30.285512 17544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a97b7cb8c8e44c088ff3ec9c6dbdeb0 [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:30.285753 17549 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:03:30.286412 17549 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:03:30.286639 17549 catalog_manager.cc:1259] Loaded cluster ID: 1d30043f004f4783a53ae82a3442b67b
I20260929 19:03:30.286835 17549 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:03:30.288123 17549 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:03:30.288297 17549 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0a97b7cb8c8e44c088ff3ec9c6dbdeb0: Loaded TSK: 0
I20260929 19:03:30.288520 17549 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 19:03:30.355605 17541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:30.355839 17541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:30.355880 17541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:30.357052 17541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:30.357121 17541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:30.357167 17541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.65
I20260929 19:03:30.358677 17541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:38173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.65.keytab
--principal=oryx/127.8.158.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.65:38095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.158.65
--webserver_port=41691
--tserver_master_addrs=127.8.158.126:40017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.65
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:30.358933 17541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:30.359165 17541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:30.365017 17568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:30.365057 17566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:30.365137 17541 server_base.cc:1034] running on GCE node
W20260929 19:03:30.365252 17565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:30.365481 17541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:30.365819 17541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:30.366990 17541 hybrid_clock.cc:648] HybridClock initialized: now 1790708610366972 us; error 43 us; skew 500 ppm
Sep 29 19:03:30 dist-test-slave-nmsz krb5kdc[16958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708610, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.158.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:30.374506 17541 init.cc:377] Logged in from keytab as oryx/127.8.158.65@KRBTEST.COM (short username oryx)
I20260929 19:03:30.375017 17541 webserver.cc:460] Webserver started at http://127.8.158.65:41691/ using document root <none> and password file <none>
I20260929 19:03:30.375250 17541 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:30.375340 17541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:30.376771 17541 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:30.377549 17575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:30.377774 17541 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:30.377877 17541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-0/wal
uuid: "511bf54a5d474c0fa73d560c2a541022"
format_stamp: "Formatted at 2026-09-29 19:03:29 on dist-test-slave-nmsz"
server_key: "e2cfeada0552229d887b0955c674cb0e"
server_key_iv: "11644758c2fca8e99bba04366546ae4d"
server_key_version: "encryptionkey@0"
I20260929 19:03:30.378298 17541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:30.402724 17541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:30.405344 17541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:30.405503 17541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:30.405756 17541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:30.406101 17541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:30.406148 17541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:30.406178 17541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:30.406195 17541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:30.413123 17541 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.65:38095
I20260929 19:03:30.413213 17688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.65:38095 every 8 connection(s)
I20260929 19:03:30.413514 17541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
I20260929 19:03:30.414749 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 17541
I20260929 19:03:30.415045 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.66:39287
--local_ip_for_outbound_sockets=127.8.158.66
--tserver_master_addrs=127.8.158.126:40017
--webserver_port=40915
--webserver_interface=127.8.158.66
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab
--principal=kudu/127.8.158.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:38173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.158.66
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 19:03:30 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.65@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:30 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.65@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:30.426846 17691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:30.414298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.65:46831)
0929 19:03:30.414499 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:30.414549 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:30.417066 (+ 2517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:30.417567 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:30.417571 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:30.418462 (+ 891us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:30.418762 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:30.418769 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:30.420439 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:30.420440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:30.421985 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:30.421988 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:30.422792 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:30.422794 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:30.422968 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:30.423584 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:30.423595 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:30.426641 (+ 3046us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":77,"threads_started":1}
W20260929 19:03:30.426934 17692 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:30.414388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:46831 (local address 127.8.158.126:40017)
0929 19:03:30.414620 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:30.414629 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:30.416421 (+ 1792us) server_negotiation.cc:407] Connection header received
0929 19:03:30.417297 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:30.417304 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:30.417357 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:30.417501 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:30.418891 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:30.420321 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:30.422085 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:30.422693 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:30.422795 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:30.426768 (+ 3973us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.65:46831: BlockingRecv error: recv got EOF from 127.8.158.65:46831 (error 108)
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":112,"threads_started":1}
W20260929 19:03:30.427465 17689 heartbeater.cc:644] Failed to heartbeat to 127.8.158.126:40017 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.158.126:40017: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Sep 29 19:03:30 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.65@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:30 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.65@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:30.432516 17691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:30.427699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.65:51719)
0929 19:03:30.427740 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:30.427752 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:30.427789 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:30.427935 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:30.427936 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:30.428085 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:30.428139 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:30.428141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:30.429021 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:30.429022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:30.429935 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:30.429937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:30.430276 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:30.430277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:30.430371 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:30.430731 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:30.430737 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:30.432474 (+ 1737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260929 19:03:30.432853 17692 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:30.427751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:51719 (local address 127.8.158.126:40017)
0929 19:03:30.427804 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:30.427806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:30.427820 (+ 14us) server_negotiation.cc:407] Connection header received
0929 19:03:30.427839 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:30.427840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:30.427850 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:30.427910 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:30.428218 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:30.428939 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:30.430015 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:30.430201 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:30.430256 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:30.432576 (+ 2320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.65:51719: BlockingRecv error: recv got EOF from 127.8.158.65:51719 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 29 19:03:30 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.65@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:30 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.65@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:30.437500 17691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:30.432758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.65:48959)
0929 19:03:30.432810 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:30.432822 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:30.432867 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:30.433147 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:30.433148 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:30.433294 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:30.433324 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:30.433325 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:30.433907 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:30.433908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:30.434862 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:30.434864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:30.435224 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:30.435225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:30.435282 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:30.435636 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:30.435641 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:30.437430 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 19:03:30.437546 17694 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:30.432850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:48959 (local address 127.8.158.126:40017)
0929 19:03:30.433015 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:30.433017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:30.433023 (+ 6us) server_negotiation.cc:407] Connection header received
0929 19:03:30.433040 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:30.433041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:30.433052 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:30.433119 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:30.433398 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:30.433819 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:30.434959 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:30.435145 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:30.435207 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:30.437506 (+ 2299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.65:48959: BlockingRecv error: recv got EOF from 127.8.158.65:48959 (error 108)
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":122,"threads_started":1}
W20260929 19:03:30.437613 17689 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 19:03:30.502111 17693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:30.502298 17693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:30.502338 17693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:30.503440 17693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:30.503476 17693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:30.503502 17693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.66
I20260929 19:03:30.504801 17693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:38173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.66.keytab
--principal=oryx/127.8.158.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.66:39287
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.158.66
--webserver_port=40915
--tserver_master_addrs=127.8.158.126:40017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.66
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:30.504962 17693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:30.505118 17693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:30.509802 17700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:30.509796 17702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:30.509802 17699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:30.510066 17693 server_base.cc:1034] running on GCE node
I20260929 19:03:30.510262 17693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:30.510531 17693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:30.511703 17693 hybrid_clock.cc:648] HybridClock initialized: now 1790708610511689 us; error 28 us; skew 500 ppm
Sep 29 19:03:30 dist-test-slave-nmsz krb5kdc[16958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708610, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.158.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:30.514304 17693 init.cc:377] Logged in from keytab as oryx/127.8.158.66@KRBTEST.COM (short username oryx)
I20260929 19:03:30.514896 17693 webserver.cc:460] Webserver started at http://127.8.158.66:40915/ using document root <none> and password file <none>
I20260929 19:03:30.515134 17693 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:30.515192 17693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:30.516675 17693 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 19:03:30.517503 17709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:30.517688 17693 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:03:30.517792 17693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-1/wal
uuid: "b1bd146f80ab4630a2001e79fb795db1"
format_stamp: "Formatted at 2026-09-29 19:03:29 on dist-test-slave-nmsz"
server_key: "9147ea1d36142929c6a08e236542d7fc"
server_key_iv: "6eff3c96e039e231d299f1edeee86536"
server_key_version: "encryptionkey@0"
I20260929 19:03:30.518159 17693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:30.551378 17693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:30.553479 17693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:30.553658 17693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:30.553893 17693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:30.554224 17693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:30.554280 17693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:30.554344 17693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:30.554375 17693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:30.560160 17693 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.66:39287
I20260929 19:03:30.560207 17822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.66:39287 every 8 connection(s)
I20260929 19:03:30.560617 17693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
Sep 29 19:03:30 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.66@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:30 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.66@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 19:03:30.567592 17694 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:30.561269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:56825 (local address 127.8.158.126:40017)
0929 19:03:30.561347 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:30.561349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:30.561589 (+ 240us) server_negotiation.cc:407] Connection header received
0929 19:03:30.562080 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:30.562081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:30.562093 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:30.562163 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:30.563071 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:30.563568 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:30.564527 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:30.564740 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:30.564825 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:30.567529 (+ 2704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.66:56825: BlockingRecv error: recv got EOF from 127.8.158.66:56825 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 19:03:30.567660 17825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:30.561270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.66:56825)
0929 19:03:30.561437 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:30.561499 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:30.561981 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:30.562232 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:30.562235 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:30.562791 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:30.562971 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:30.562975 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:30.563661 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:30.563662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:30.564418 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:30.564420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:30.564834 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:30.564835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:30.564948 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:30.565349 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:30.565356 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:30.567468 (+ 2112us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":90,"threads_started":1}
W20260929 19:03:30.567888 17823 heartbeater.cc:644] Failed to heartbeat to 127.8.158.126:40017 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.158.126:40017: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
I20260929 19:03:30.570835 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 17693
Sep 29 19:03:30 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.66@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:30.571112 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.67:46129
--local_ip_for_outbound_sockets=127.8.158.67
--tserver_master_addrs=127.8.158.126:40017
--webserver_port=40603
--webserver_interface=127.8.158.67
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab
--principal=kudu/127.8.158.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:38173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.158.67
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 19:03:30 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.66@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:30.572176 17825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:30.568091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.66:46071)
0929 19:03:30.568140 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:30.568151 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:30.568191 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:30.568306 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:30.568307 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:30.568431 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:30.568469 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:30.568470 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:30.569061 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:30.569063 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:30.569562 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:30.569563 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:30.569904 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:30.569905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:30.569974 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:30.570312 (+ 338us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:30.570350 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:30.572081 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 19:03:30.572309 17694 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:30.568187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:46071 (local address 127.8.158.126:40017)
0929 19:03:30.568221 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:30.568223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:30.568230 (+ 7us) server_negotiation.cc:407] Connection header received
0929 19:03:30.568249 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:30.568250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:30.568258 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:30.568305 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:30.568550 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:30.568987 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:30.569656 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:30.569836 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:30.569908 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:30.572201 (+ 2293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.66:46071: BlockingRecv error: recv got EOF from 127.8.158.66:46071 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
Sep 29 19:03:30 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.66@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:30 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.66@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:30.576553 17825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:30.572406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.66:51743)
0929 19:03:30.572523 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:30.572534 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:30.572577 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:30.572691 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:30.572692 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:30.572804 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:30.572825 (+ 21us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:30.572826 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:30.573431 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:30.573432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:30.573902 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:30.573903 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:30.574396 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:30.574397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:30.574433 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:30.574819 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:30.574823 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:30.576464 (+ 1641us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105}
W20260929 19:03:30.576561 17694 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:30.572488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:51743 (local address 127.8.158.126:40017)
0929 19:03:30.572526 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:30.572528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:30.572563 (+ 35us) server_negotiation.cc:407] Connection header received
0929 19:03:30.572631 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:30.572632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:30.572641 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:30.572680 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:30.572898 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:30.573354 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:30.573985 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:30.574295 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:30.574384 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:30.576525 (+ 2141us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.66:51743: BlockingRecv error: recv got EOF from 127.8.158.66:51743 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260929 19:03:30.576668 17823 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 19:03:30.651147 17826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:30.651329 17826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:30.651350 17826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:30.652519 17826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:30.652560 17826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:30.652586 17826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.67
I20260929 19:03:30.653981 17826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:38173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/krb5kdc/oryx_127.8.158.67.keytab
--principal=oryx/127.8.158.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.67:46129
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.158.67
--webserver_port=40603
--tserver_master_addrs=127.8.158.126:40017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.67
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:30.654156 17826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:30.654393 17826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:30.659160 17831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:30.659505 17826 server_base.cc:1034] running on GCE node
W20260929 19:03:30.659150 17834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:30.659152 17832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:30.659809 17826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:30.660095 17826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:30.661257 17826 hybrid_clock.cc:648] HybridClock initialized: now 1790708610661248 us; error 33 us; skew 500 ppm
Sep 29 19:03:30 dist-test-slave-nmsz krb5kdc[16958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708610, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.158.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:30.664172 17826 init.cc:377] Logged in from keytab as oryx/127.8.158.67@KRBTEST.COM (short username oryx)
I20260929 19:03:30.664707 17826 webserver.cc:460] Webserver started at http://127.8.158.67:40603/ using document root <none> and password file <none>
I20260929 19:03:30.664958 17826 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:30.665045 17826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:30.666559 17826 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:03:30.667132 17841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:30.667296 17826 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:30.667434 17826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-2/wal
uuid: "7e17cac163394108be87d5e2932e97bc"
format_stamp: "Formatted at 2026-09-29 19:03:29 on dist-test-slave-nmsz"
server_key: "4d833949c15b2f682144dcc276c04f22"
server_key_iv: "2637ae36d63e55d6d314ded0bda2b4ef"
server_key_version: "encryptionkey@0"
I20260929 19:03:30.667788 17826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:30.696604 17826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:30.698804 17826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:30.698976 17826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:30.699221 17826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:30.699542 17826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:30.699600 17826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:30.699644 17826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:30.699676 17826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:30.705870 17826 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.67:46129
I20260929 19:03:30.705917 17954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.67:46129 every 8 connection(s)
I20260929 19:03:30.706367 17826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
Sep 29 19:03:30 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.67@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:30 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.67@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 19:03:30.713886 17694 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:30.707118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:53539 (local address 127.8.158.126:40017)
0929 19:03:30.707164 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:30.707166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:30.707406 (+ 240us) server_negotiation.cc:407] Connection header received
0929 19:03:30.707868 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:30.707869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:30.707879 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:30.707941 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:30.708814 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:30.709383 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:30.710429 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:30.710624 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:30.710684 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:30.713827 (+ 3143us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.67:53539: BlockingRecv error: recv got EOF from 127.8.158.67:53539 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 19:03:30.713954 17957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:30.707126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.67:53539)
0929 19:03:30.707284 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:30.707317 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:30.707773 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:30.708006 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:30.708010 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:30.708554 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:30.708733 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:30.708737 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:30.709469 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:30.709470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:30.710239 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:30.710242 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:30.710701 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:30.710702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:30.710806 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:30.711345 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:30.711353 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:30.713736 (+ 2383us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":81,"threads_started":1}
W20260929 19:03:30.714212 17955 heartbeater.cc:644] Failed to heartbeat to 127.8.158.126:40017 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.158.126:40017: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
I20260929 19:03:30.716375 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 17826
Sep 29 19:03:30 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.67@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:30 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.67@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:30.719170 17957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:30.714491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.67:37931)
0929 19:03:30.714545 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:30.714557 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:30.714606 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:30.714853 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:30.714853 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:30.714991 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:30.715039 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:30.715040 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:30.715738 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:30.715739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:30.716303 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:30.716305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:30.716773 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:30.716774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:30.716849 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:30.717334 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:30.717338 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:30.719107 (+ 1769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 19:03:30.719300 17694 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:30.714687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:37931 (local address 127.8.158.126:40017)
0929 19:03:30.714721 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:30.714723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:30.714736 (+ 13us) server_negotiation.cc:407] Connection header received
0929 19:03:30.714755 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:30.714756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:30.714763 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:30.714850 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:30.715142 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:30.715650 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:30.716391 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:30.716657 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:30.716740 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:30.719196 (+ 2456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.67:37931: BlockingRecv error: recv got EOF from 127.8.158.67:37931 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
Sep 29 19:03:30 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.67@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:30 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.67@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:30.724603 17957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:30.719397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.67:54207)
0929 19:03:30.719747 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:30.719755 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:30.719809 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:30.719919 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:30.719920 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:30.720062 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:30.720089 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:30.720091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:30.720924 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:30.720926 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:30.721646 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:30.721648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:30.722077 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:30.722078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:30.722136 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:30.722634 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:30.722639 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:30.724545 (+ 1906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":338}
W20260929 19:03:30.724692 17694 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:30.719545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:54207 (local address 127.8.158.126:40017)
0929 19:03:30.719606 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:30.719607 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:30.719798 (+ 191us) server_negotiation.cc:407] Connection header received
0929 19:03:30.719856 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:30.719857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:30.719866 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:30.719918 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:30.720185 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:30.720841 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:30.721697 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:30.722002 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:30.722057 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:30.724619 (+ 2562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.67:54207: BlockingRecv error: recv got EOF from 127.8.158.67:54207 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260929 19:03:30.724709 17955 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 19:03:30.724949 17692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:30.716789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46282 (local address 127.8.158.126:40017)
0929 19:03:30.716853 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:30.716855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:30.716950 (+ 95us) server_negotiation.cc:407] Connection header received
0929 19:03:30.716975 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:30.716975 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:30.716989 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:30.717048 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:30.717717 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:30.718442 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:30.719756 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:30.719956 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:30.720019 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:30.721401 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:30.721405 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:30.721406 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:30.721416 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:30.723402 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:30.723853 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:30.723857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:30.723858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:30.723981 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:30.724183 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:30.724184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:30.724185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:30.724453 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:30.724615 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:30.724779 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 19:03:30.724892 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Sep 29 19:03:31 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.65@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:31 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.65@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:31.443219 17959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:31.437981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.65:34483)
0929 19:03:31.438156 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:31.438168 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:31.438235 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:31.438463 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:31.438464 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:31.438628 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:31.438673 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:31.438675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:31.439405 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:31.439406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:31.439992 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:31.439994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:31.440391 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:31.440392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:31.440434 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:31.440965 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:31.440969 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:31.443112 (+ 2143us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":90,"threads_started":1}
W20260929 19:03:31.443284 17960 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:31.438074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:34483 (local address 127.8.158.126:40017)
0929 19:03:31.438182 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:31.438184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:31.438223 (+ 39us) server_negotiation.cc:407] Connection header received
0929 19:03:31.438341 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:31.438342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:31.438354 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:31.438451 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:31.438751 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:31.439328 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:31.440073 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:31.440318 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:31.440379 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:31.443186 (+ 2807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.65:34483: BlockingRecv error: recv got EOF from 127.8.158.65:34483 (error 108)
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":56,"threads_started":1}
Sep 29 19:03:31 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.66@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:31 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.66@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:31.581811 17961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:31.577042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.66:50531)
0929 19:03:31.577167 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:31.577184 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:31.577231 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:31.577369 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:31.577370 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:31.577522 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:31.577569 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:31.577570 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:31.578214 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:31.578215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:31.578793 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:31.578795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:31.579150 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:31.579151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:31.579193 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:31.579651 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:31.579655 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:31.581744 (+ 2089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":81,"threads_started":1}
W20260929 19:03:31.581925 17960 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:31.577116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:50531 (local address 127.8.158.126:40017)
0929 19:03:31.577199 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:31.577201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:31.577218 (+ 17us) server_negotiation.cc:407] Connection header received
0929 19:03:31.577287 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:31.577287 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:31.577298 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:31.577354 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:31.577633 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:31.578125 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:31.578882 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:31.579078 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:31.579136 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:31.581850 (+ 2714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.66:50531: BlockingRecv error: recv got EOF from 127.8.158.66:50531 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 29 19:03:31 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.67@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:31 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.67@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:31.729877 17962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:31.725062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.67:36519)
0929 19:03:31.725210 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:31.725220 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:31.725275 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:31.725425 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:31.725425 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:31.725578 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:31.725625 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:31.725627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:31.726227 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:31.726227 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:31.726817 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:31.726819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:31.727145 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:31.727146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:31.727191 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:31.727688 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:31.727692 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:31.729811 (+ 2119us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":89,"threads_started":1}
W20260929 19:03:31.729990 17960 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:31.725138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:36519 (local address 127.8.158.126:40017)
0929 19:03:31.725214 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:31.725215 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:31.725278 (+ 63us) server_negotiation.cc:407] Connection header received
0929 19:03:31.725332 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:31.725333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:31.725345 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:31.725402 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:31.725689 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:31.726163 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:31.726898 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:31.727076 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:31.727131 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:31.729900 (+ 2769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.67:36519: BlockingRecv error: recv got EOF from 127.8.158.67:36519 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 29 19:03:32 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.65@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:32 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.65@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:32.448239 17963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:32.443723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.65:56539)
0929 19:03:32.443876 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:32.443886 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:32.443954 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:32.444096 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:32.444097 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:32.444273 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:32.444317 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:32.444319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:32.444975 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:32.444976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:32.445588 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:32.445591 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:32.445994 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:32.445995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:32.446042 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:32.446494 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:32.446498 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:32.448174 (+ 1676us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":100,"threads_started":1}
W20260929 19:03:32.448436 17964 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:32.443801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:56539 (local address 127.8.158.126:40017)
0929 19:03:32.443910 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:32.443911 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:32.443925 (+ 14us) server_negotiation.cc:407] Connection header received
0929 19:03:32.444027 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:32.444027 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:32.444036 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:32.444090 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:32.444378 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:32.444883 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:32.445674 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:32.445908 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:32.445966 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:32.448346 (+ 2380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.65:56539: BlockingRecv error: recv got EOF from 127.8.158.65:56539 (error 108)
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":63,"threads_started":1}
Sep 29 19:03:32 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.66@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:32 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.66@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:32.587633 17965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:32.582429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.66:46943)
0929 19:03:32.582604 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:32.582616 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:32.582688 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:32.582872 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:32.582873 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:32.583061 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:32.583105 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:32.583107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:32.583890 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:32.583891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:32.584465 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:32.584466 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:32.584864 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:32.584865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:32.584959 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:32.585519 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:32.585524 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:32.587539 (+ 2015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":99,"threads_started":1}
W20260929 19:03:32.587673 17964 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:32.582498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:46943 (local address 127.8.158.126:40017)
0929 19:03:32.582593 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:32.582596 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:32.582667 (+ 71us) server_negotiation.cc:407] Connection header received
0929 19:03:32.582796 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:32.582796 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:32.582809 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:32.582865 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:32.583191 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:32.583809 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:32.584545 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:32.584779 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:32.584843 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:32.587614 (+ 2771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.66:46943: BlockingRecv error: recv got EOF from 127.8.158.66:46943 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 29 19:03:32 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.67@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:32 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.67@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:32.735782 17966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:32.730379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.67:45805)
0929 19:03:32.730599 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:32.730612 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:32.730716 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:32.730916 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:32.730917 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:32.731105 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:32.731151 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:32.731154 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:32.732021 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:32.732023 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:32.732629 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:32.732631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:32.733029 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:32.733030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:32.733095 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:32.733583 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:32.733588 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:32.735676 (+ 2088us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":101,"threads_started":1}
W20260929 19:03:32.735837 17964 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:32.730487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:45805 (local address 127.8.158.126:40017)
0929 19:03:32.730592 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:32.730595 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:32.730695 (+ 100us) server_negotiation.cc:407] Connection header received
0929 19:03:32.730787 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:32.730788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:32.730802 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:32.730886 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:32.731255 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:32.731927 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:32.732740 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:32.732948 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:32.733031 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:32.735766 (+ 2735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.67:45805: BlockingRecv error: recv got EOF from 127.8.158.67:45805 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 29 19:03:33 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.65@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:33 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.65@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:33.453622 17967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:33.448688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.65:53079)
0929 19:03:33.448862 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:33.448878 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:33.448927 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:33.449087 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:33.449088 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:33.449279 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:33.449468 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:33.449471 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:33.450346 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:33.450347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:33.450865 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:33.450867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:33.451256 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:33.451257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:33.451318 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:33.451810 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:33.451815 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:33.453566 (+ 1751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":124,"threads_started":1}
W20260929 19:03:33.453742 17968 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:33.448779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:53079 (local address 127.8.158.126:40017)
0929 19:03:33.448912 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:33.448914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:33.448923 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:03:33.448989 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:33.448989 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:33.449000 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:33.449055 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:33.449578 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:33.450238 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:33.450963 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:33.451163 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:33.451235 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:33.453663 (+ 2428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.65:53079: BlockingRecv error: recv got EOF from 127.8.158.65:53079 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":69,"threads_started":1}
Sep 29 19:03:33 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.66@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:33 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.66@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:33.593024 17969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:33.588144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.66:54699)
0929 19:03:33.588326 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:33.588345 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:33.588402 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:33.588547 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:33.588548 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:33.588709 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:33.588764 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:33.588765 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:33.589535 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:33.589535 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:33.590064 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:33.590066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:33.590446 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:33.590447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:33.590491 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:33.590949 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:33.590953 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:33.592929 (+ 1976us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":133,"threads_started":1}
W20260929 19:03:33.593071 17968 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:33.588242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:54699 (local address 127.8.158.126:40017)
0929 19:03:33.588329 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:33.588332 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:33.588409 (+ 77us) server_negotiation.cc:407] Connection header received
0929 19:03:33.588443 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:33.588444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:33.588455 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:33.588531 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:33.588838 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:33.589440 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:33.590160 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:33.590380 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:33.590443 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:33.593009 (+ 2566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.66:54699: BlockingRecv error: recv got EOF from 127.8.158.66:54699 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 29 19:03:33 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.67@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:33 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.67@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:33.741305 17970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:33.736354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.67:51565)
0929 19:03:33.736608 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:33.736623 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:33.736699 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:33.736906 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:33.736907 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:33.737071 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:33.737119 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:33.737121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:33.737829 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:33.737830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:33.738394 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:33.738396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:33.738756 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:33.738757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:33.738804 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:33.739325 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:33.739330 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:33.741247 (+ 1917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":69,"threads_started":1}
W20260929 19:03:33.741480 17968 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:33.736548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:51565 (local address 127.8.158.126:40017)
0929 19:03:33.736616 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:33.736617 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:33.736746 (+ 129us) server_negotiation.cc:407] Connection header received
0929 19:03:33.736773 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:33.736774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:33.736793 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:33.736860 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:33.737203 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:33.737744 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:33.738491 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:33.738675 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:33.738751 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:33.741384 (+ 2633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.67:51565: BlockingRecv error: recv got EOF from 127.8.158.67:51565 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 29 19:03:34 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.65@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:34 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.65@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:34.459239 17971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:34.454120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.65:55061)
0929 19:03:34.454273 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:34.454284 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:34.454409 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:34.454539 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:34.454539 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:34.454694 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:34.454747 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:34.454749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:34.455371 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:34.455372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:34.455901 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:34.455903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:34.456226 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:34.456226 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:34.456265 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:34.456698 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:34.456702 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:34.459182 (+ 2480us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":136,"threads_started":1}
W20260929 19:03:34.459420 17972 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:34.454193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:55061 (local address 127.8.158.126:40017)
0929 19:03:34.454297 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:34.454299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:34.454379 (+ 80us) server_negotiation.cc:407] Connection header received
0929 19:03:34.454453 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:34.454454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:34.454464 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:34.454519 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:34.454807 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:34.455301 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:34.455972 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:34.456160 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:34.456216 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:34.459337 (+ 3121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.65:55061: BlockingRecv error: recv got EOF from 127.8.158.65:55061 (error 108)
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":48,"threads_started":1}
Sep 29 19:03:34 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.66@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:34 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.66@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:34.598129 17973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:34.593651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.66:52457)
0929 19:03:34.593760 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:34.593773 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:34.593840 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:34.594011 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:34.594013 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:34.594217 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:34.594261 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:34.594263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:34.594971 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:34.594972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:34.595495 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:34.595496 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:34.595855 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:34.595856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:34.595913 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:34.596370 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:34.596374 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:34.598032 (+ 1658us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":60,"threads_started":1}
W20260929 19:03:34.598165 17972 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:34.593713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:52457 (local address 127.8.158.126:40017)
0929 19:03:34.593757 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:34.593760 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:34.593852 (+ 92us) server_negotiation.cc:407] Connection header received
0929 19:03:34.593907 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:34.593907 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:34.593921 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:34.594000 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:34.594383 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:34.594886 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:34.595592 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:34.595783 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:34.595859 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:34.598105 (+ 2246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.66:52457: BlockingRecv error: recv got EOF from 127.8.158.66:52457 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 29 19:03:34 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.67@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:34 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.67@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:34.747413 17974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:34.741967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.67:45851)
0929 19:03:34.742113 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:34.742132 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:34.742193 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:34.742350 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:34.742351 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:34.742721 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:34.742766 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:34.742768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:34.743502 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:34.743502 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:34.744104 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:34.744107 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:34.744560 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:34.744579 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:34.744647 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:34.745123 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:34.745127 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:34.747313 (+ 2186us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":94,"threads_started":1}
W20260929 19:03:34.747457 17972 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:34.742058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:45851 (local address 127.8.158.126:40017)
0929 19:03:34.742143 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:34.742145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:34.742201 (+ 56us) server_negotiation.cc:407] Connection header received
0929 19:03:34.742231 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:34.742232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:34.742243 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:34.742327 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:34.742858 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:34.743393 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:34.744240 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:34.744485 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:34.744549 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:34.747391 (+ 2842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.67:45851: BlockingRecv error: recv got EOF from 127.8.158.67:45851 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 19:03:35 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.65@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:35 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.65@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:35.469534 17975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:35.459765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.65:47411)
0929 19:03:35.459903 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:35.459913 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:35.459961 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:35.460137 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:35.460139 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:35.460304 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:35.460350 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:35.460352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:35.461048 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:35.461049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:35.461642 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:35.461643 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:35.462013 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:35.462014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:35.462072 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:35.467490 (+ 5418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:35.467495 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:35.469439 (+ 1944us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":79,"threads_started":1}
W20260929 19:03:35.469584 17976 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:35.459854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:47411 (local address 127.8.158.126:40017)
0929 19:03:35.459979 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:35.459983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:35.459996 (+ 13us) server_negotiation.cc:407] Connection header received
0929 19:03:35.460016 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:35.460016 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:35.460026 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:35.460091 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:35.460458 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:35.460950 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:35.461744 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:35.461934 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:35.461992 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:35.469515 (+ 7523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.65:47411: BlockingRecv error: recv got EOF from 127.8.158.65:47411 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":76,"threads_started":1}
Sep 29 19:03:35 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.66@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:35 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.66@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:35.603364 17977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:35.598588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.66:58091)
0929 19:03:35.598714 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:35.598726 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:35.598786 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:35.598937 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:35.598938 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:35.599121 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:35.599201 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:35.599203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:35.599858 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:35.599860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:35.600384 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:35.600385 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:35.600754 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:35.600756 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:35.600816 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:35.601339 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:35.601343 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:35.603303 (+ 1960us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":95,"threads_started":1}
W20260929 19:03:35.603556 17976 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:35.598762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:58091 (local address 127.8.158.126:40017)
0929 19:03:35.598811 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:35.598813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:35.598822 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:03:35.598848 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:35.598848 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:35.598859 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:35.598931 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:35.599285 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:35.599776 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:35.600469 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:35.600679 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:35.600756 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:35.603457 (+ 2701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.66:58091: BlockingRecv error: recv got EOF from 127.8.158.66:58091 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 29 19:03:35 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.67@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:35 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.67@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:35.752791 17978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:35.747955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.67:57049)
0929 19:03:35.748136 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:35.748147 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:35.748239 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:35.748411 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:35.748412 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:35.748590 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:35.748633 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:35.748635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:35.749354 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:35.749355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:35.750092 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:35.750095 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:35.750486 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:35.750487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:35.750544 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:35.750972 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:35.750977 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:35.752696 (+ 1719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":94,"threads_started":1}
W20260929 19:03:35.752821 17976 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:35.748046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:57049 (local address 127.8.158.126:40017)
0929 19:03:35.748126 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:35.748129 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:35.748222 (+ 93us) server_negotiation.cc:407] Connection header received
0929 19:03:35.748325 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:35.748325 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:35.748337 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:35.748406 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:35.748735 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:35.749265 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:35.750158 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:35.750405 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:35.750486 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:35.752765 (+ 2279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.67:57049: BlockingRecv error: recv got EOF from 127.8.158.67:57049 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 19:03:36 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.65@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:36 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.65@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:36.474862 17979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:36.470087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.65:39665)
0929 19:03:36.470221 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:36.470240 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:36.470297 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:36.470507 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:36.470507 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:36.470675 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:36.470722 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:36.470723 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:36.471406 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:36.471407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:36.471995 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:36.471997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:36.472315 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:36.472316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:36.472351 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:36.472934 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:36.472939 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:36.474797 (+ 1858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":56,"threads_started":1}
W20260929 19:03:36.475196 17980 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:36.470240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:39665 (local address 127.8.158.126:40017)
0929 19:03:36.470382 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:36.470384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:36.470392 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:03:36.470421 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:36.470422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:36.470432 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:36.470488 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:36.470787 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:36.471341 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:36.472061 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:36.472252 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:36.472306 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:36.475012 (+ 2706us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.65:39665: BlockingRecv error: recv got EOF from 127.8.158.65:39665 (error 108)
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
Sep 29 19:03:36 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.66@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:36 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.66@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:36.608985 17981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:36.604024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.66:34005)
0929 19:03:36.604216 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:36.604227 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:36.604298 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:36.604497 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:36.604499 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:36.604653 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:36.604700 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:36.604702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:36.605362 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:36.605363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:36.606061 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:36.606063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:36.606630 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:36.606630 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:36.606675 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:36.607163 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:36.607168 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:36.608912 (+ 1744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":99,"threads_started":1}
W20260929 19:03:36.609136 17980 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:36.604093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:34005 (local address 127.8.158.126:40017)
0929 19:03:36.604176 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:36.604178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:36.604332 (+ 154us) server_negotiation.cc:407] Connection header received
0929 19:03:36.604362 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:36.604363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:36.604381 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:36.604442 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:36.604797 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:36.605283 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:36.606148 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:36.606520 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:36.606594 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:36.609034 (+ 2440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.66:34005: BlockingRecv error: recv got EOF from 127.8.158.66:34005 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 19:03:36 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.67@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:36 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.67@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:36.758188 17982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:36.753325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.67:51823)
0929 19:03:36.753490 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:36.753505 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:36.753575 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:36.753701 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:36.753702 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:36.753855 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:36.753902 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:36.753903 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:36.754578 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:36.754579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:36.755113 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:36.755115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:36.755444 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:36.755445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:36.755486 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:36.755949 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:36.755954 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:36.758098 (+ 2144us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":64,"threads_started":1}
W20260929 19:03:36.758280 17980 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:36.753445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:51823 (local address 127.8.158.126:40017)
0929 19:03:36.753531 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:36.753533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:36.753561 (+ 28us) server_negotiation.cc:407] Connection header received
0929 19:03:36.753626 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:36.753627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:36.753638 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:36.753678 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:36.753967 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:36.754491 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:36.755199 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:36.755375 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:36.755428 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:36.758211 (+ 2783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.67:51823: BlockingRecv error: recv got EOF from 127.8.158.67:51823 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 19:03:37 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.65@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:37 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.65@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:37.480031 17983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:37.475357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.65:53971)
0929 19:03:37.475509 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:37.475519 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:37.475570 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:37.475737 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:37.475738 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:37.475891 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:37.475938 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:37.475940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:37.476600 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:37.476601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:37.477120 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:37.477121 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:37.477520 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:37.477520 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:37.477559 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:37.478032 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:37.478036 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:37.479936 (+ 1900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":77,"threads_started":1}
W20260929 19:03:37.480080 17984 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:37.475458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:53971 (local address 127.8.158.126:40017)
0929 19:03:37.475608 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:37.475610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:37.475624 (+ 14us) server_negotiation.cc:407] Connection header received
0929 19:03:37.475644 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:37.475644 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:37.475655 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:37.475718 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:37.476018 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:37.476526 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:37.477206 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:37.477429 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:37.477508 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:37.480011 (+ 2503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.65:53971: BlockingRecv error: recv got EOF from 127.8.158.65:53971 (error 108)
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":94,"threads_started":1}
Sep 29 19:03:37 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.66@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:37 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.66@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:37.614920 17985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:37.609547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.66:45945)
0929 19:03:37.609763 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:37.609777 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:37.609870 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:37.610030 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:37.610030 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:37.610221 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:37.610267 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:37.610268 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:37.611042 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:37.611043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:37.611622 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:37.611623 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:37.612048 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:37.612049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:37.612114 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:37.612731 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:37.612738 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:37.614861 (+ 2123us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":110,"threads_started":1}
W20260929 19:03:37.615067 17984 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:37.609659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:45945 (local address 127.8.158.126:40017)
0929 19:03:37.609747 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:37.609749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:37.609848 (+ 99us) server_negotiation.cc:407] Connection header received
0929 19:03:37.609954 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:37.609955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:37.609969 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:37.610021 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:37.610393 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:37.610940 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:37.611723 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:37.611939 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:37.612035 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:37.614983 (+ 2948us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.66:45945: BlockingRecv error: recv got EOF from 127.8.158.66:45945 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 29 19:03:37 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.67@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:37 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.67@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:37.763845 17986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:37.758841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.67:50365)
0929 19:03:37.758994 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:37.759004 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:37.759063 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:37.759203 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:37.759204 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:37.759374 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:37.759426 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:37.759428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:37.760104 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:37.760106 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:37.760682 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:37.760683 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:37.761079 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:37.761080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:37.761134 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:37.761669 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:37.761674 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:37.763770 (+ 2096us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":67,"threads_started":1}
W20260929 19:03:37.764070 17984 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:37.758901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:50365 (local address 127.8.158.126:40017)
0929 19:03:37.759003 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:37.759005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:37.759074 (+ 69us) server_negotiation.cc:407] Connection header received
0929 19:03:37.759101 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:37.759101 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:37.759114 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:37.759172 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:37.759494 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:37.760017 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:37.760787 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:37.760982 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:37.761044 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:37.764003 (+ 2959us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.67:50365: BlockingRecv error: recv got EOF from 127.8.158.67:50365 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 29 19:03:38 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.65@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:38 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.65@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:38.485663 17987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:38.480555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.65:44433)
0929 19:03:38.480692 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:38.480703 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:38.480751 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:38.481032 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:38.481033 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:38.481203 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:38.481252 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:38.481254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:38.481938 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:38.481939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:38.482557 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:38.482559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:38.482967 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:38.482968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:38.483017 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:38.483614 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:38.483619 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:38.485573 (+ 1954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
W20260929 19:03:38.485817 17988 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:38.480715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:44433 (local address 127.8.158.126:40017)
0929 19:03:38.480850 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:38.480851 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:38.480858 (+ 7us) server_negotiation.cc:407] Connection header received
0929 19:03:38.480878 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:38.480878 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:38.480886 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:38.480959 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:38.481340 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:38.481853 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:38.482674 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:38.482877 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:38.482943 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:38.485719 (+ 2776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.65:44433: BlockingRecv error: recv got EOF from 127.8.158.65:44433 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":82,"threads_started":1}
Sep 29 19:03:38 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.66@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:38 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.66@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:38.620692 17989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:38.615514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.66:49493)
0929 19:03:38.615688 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:38.615702 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:38.615763 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:38.615967 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:38.615968 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:38.616240 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:38.616297 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:38.616299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:38.617229 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:38.617230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:38.617805 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:38.617807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:38.618184 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:38.618185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:38.618229 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:38.618780 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:38.618784 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:38.620623 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":124,"threads_started":1}
W20260929 19:03:38.620788 17988 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:38.615618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:49493 (local address 127.8.158.126:40017)
0929 19:03:38.615773 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:38.615775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:38.615804 (+ 29us) server_negotiation.cc:407] Connection header received
0929 19:03:38.615832 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:38.615833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:38.615842 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:38.615924 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:38.616394 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:38.617130 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:38.617888 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:38.618109 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:38.618170 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:38.620719 (+ 2549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.66:49493: BlockingRecv error: recv got EOF from 127.8.158.66:49493 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 29 19:03:38 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.67@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:38 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.67@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:38.769068 17990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:38.764454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.67:40915)
0929 19:03:38.764597 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:38.764608 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:38.764676 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:38.764796 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:38.764796 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:38.764986 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:38.765031 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:38.765033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:38.765654 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:38.765655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:38.766189 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:38.766190 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:38.766605 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:38.766606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:38.766654 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:38.767101 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:38.767105 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:38.769011 (+ 1906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":89,"threads_started":1}
W20260929 19:03:38.769273 17988 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:38.764533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:40915 (local address 127.8.158.126:40017)
0929 19:03:38.764606 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:38.764608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:38.764661 (+ 53us) server_negotiation.cc:407] Connection header received
0929 19:03:38.764719 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:38.764719 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:38.764731 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:38.764791 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:38.765093 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:38.765585 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:38.766261 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:38.766508 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:38.766570 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:38.769198 (+ 2628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.67:40915: BlockingRecv error: recv got EOF from 127.8.158.67:40915 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 29 19:03:39 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.65@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:39 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.65@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:39.491182 17991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:39.486099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.65:33031)
0929 19:03:39.486423 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:39.486436 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:39.486515 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:39.486675 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:39.486676 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:39.486921 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:39.486969 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:39.486971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:39.487623 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:39.487624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:39.488158 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:39.488160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:39.488508 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:39.488508 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:39.488546 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:39.489028 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:39.489033 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:39.491113 (+ 2080us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":186,"threads_started":1}
W20260929 19:03:39.491328 17992 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:39.486291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:33031 (local address 127.8.158.126:40017)
0929 19:03:39.486437 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:39.486438 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:39.486522 (+ 84us) server_negotiation.cc:407] Connection header received
0929 19:03:39.486573 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:39.486574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:39.486584 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:39.486645 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:39.487040 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:39.487547 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:39.488234 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:39.488441 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:39.488495 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:39.491249 (+ 2754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.65:33031: BlockingRecv error: recv got EOF from 127.8.158.65:33031 (error 108)
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":75,"threads_started":1}
Sep 29 19:03:39 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.66@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:39 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.66@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:39.626224 17993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:39.621221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.66:48109)
0929 19:03:39.621410 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:39.621421 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:39.621513 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:39.621678 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:39.621679 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:39.621862 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:39.621906 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:39.621908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:39.622690 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:39.622691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:39.623254 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:39.623256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:39.623630 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:39.623631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:39.623690 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:39.624126 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:39.624130 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:39.626130 (+ 2000us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":103,"threads_started":1}
W20260929 19:03:39.626273 17992 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:39.621326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:48109 (local address 127.8.158.126:40017)
0929 19:03:39.621376 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:39.621378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:39.621494 (+ 116us) server_negotiation.cc:407] Connection header received
0929 19:03:39.621579 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:39.621580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:39.621592 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:39.621673 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:39.621989 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:39.622594 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:39.623355 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:39.623547 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:39.623620 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:39.626209 (+ 2589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.66:48109: BlockingRecv error: recv got EOF from 127.8.158.66:48109 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 29 19:03:39 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.67@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:39 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.67@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:39.774931 17994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:39.769706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.67:53845)
0929 19:03:39.769879 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:39.769900 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:39.769960 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:39.770123 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:39.770124 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:39.770294 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:39.770421 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:39.770423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:39.771203 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:39.771204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:39.771885 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:39.771887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:39.772269 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:39.772270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:39.772316 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:39.772858 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:39.772863 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:39.774861 (+ 1998us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":72,"threads_started":1}
W20260929 19:03:39.775012 17992 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:39.769838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:53845 (local address 127.8.158.126:40017)
0929 19:03:39.769943 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:39.769945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:39.769956 (+ 11us) server_negotiation.cc:407] Connection header received
0929 19:03:39.770027 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:39.770028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:39.770043 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:39.770096 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:39.770511 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:39.771127 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:39.771986 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:39.772185 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:39.772253 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:39.774955 (+ 2702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.67:53845: BlockingRecv error: recv got EOF from 127.8.158.67:53845 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 29 19:03:40 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.65@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:40 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.65@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:40.501327 17995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:40.491617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.65:46921)
0929 19:03:40.491761 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:40.491776 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:40.491825 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:40.491996 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:40.491997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:40.492156 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:40.492200 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:40.492202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:40.492901 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:40.492902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:40.493434 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:40.493436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:40.494025 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:40.494027 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:40.494206 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:40.499514 (+ 5308us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:40.499520 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:40.501269 (+ 1749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":86,"threads_started":1}
W20260929 19:03:40.501513 17996 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:40.491723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:46921 (local address 127.8.158.126:40017)
0929 19:03:40.491856 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:40.491860 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:40.491872 (+ 12us) server_negotiation.cc:407] Connection header received
0929 19:03:40.491893 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:40.491893 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:40.491903 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:40.491969 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:40.492283 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:40.492773 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:40.493603 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:40.493849 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:40.493953 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:40.501410 (+ 7457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.65:46921: BlockingRecv error: recv got EOF from 127.8.158.65:46921 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":80,"threads_started":1}
Sep 29 19:03:40 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.66@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:40 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.66@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:40.631875 17997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:40.626724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.66:52977)
0929 19:03:40.626919 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:40.626932 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:40.627026 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:40.627202 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:40.627202 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:40.627391 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:40.627435 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:40.627437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:40.628204 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:40.628205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:40.628944 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:40.628947 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:40.629320 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:40.629321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:40.629375 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:40.629855 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:40.629860 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:40.631803 (+ 1943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":100,"threads_started":1}
W20260929 19:03:40.632011 17996 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:40.626861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:52977 (local address 127.8.158.126:40017)
0929 19:03:40.626934 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:40.626936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:40.627011 (+ 75us) server_negotiation.cc:407] Connection header received
0929 19:03:40.627088 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:40.627089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:40.627100 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:40.627181 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:40.627517 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:40.628104 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:40.629028 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:40.629240 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:40.629324 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:40.631922 (+ 2598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.66:52977: BlockingRecv error: recv got EOF from 127.8.158.66:52977 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 29 19:03:40 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.67@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:40 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.67@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:40.780269 17998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:40.775406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.67:41115)
0929 19:03:40.775563 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:40.775578 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:40.775650 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:40.775801 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:40.775802 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:40.775982 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:40.776030 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:40.776031 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:40.776677 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:40.776678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:40.777207 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:40.777209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:40.777564 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:40.777564 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:40.777625 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:40.778069 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:40.778074 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:40.780170 (+ 2096us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
W20260929 19:03:40.780296 17996 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:40.775523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:41115 (local address 127.8.158.126:40017)
0929 19:03:40.775595 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:40.775597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:40.775631 (+ 34us) server_negotiation.cc:407] Connection header received
0929 19:03:40.775713 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:40.775714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:40.775732 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:40.775795 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:40.776109 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:40.776593 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:40.777291 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:40.777477 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:40.777561 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:40.780238 (+ 2677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.67:41115: BlockingRecv error: recv got EOF from 127.8.158.67:41115 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 29 19:03:41 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.65@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:41 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.65@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:41.507089 17999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:41.501828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.65:55347)
0929 19:03:41.502013 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:41.502024 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:41.502107 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:41.502429 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:41.502431 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:41.502703 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:41.502759 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:41.502762 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:41.503527 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:41.503528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:41.504130 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:41.504132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:41.504490 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:41.504491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:41.504549 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:41.505016 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:41.505021 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:41.507020 (+ 1999us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":71,"threads_started":1}
W20260929 19:03:41.507225 18000 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:41.501950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:55347 (local address 127.8.158.126:40017)
0929 19:03:41.502076 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:41.502078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:41.502089 (+ 11us) server_negotiation.cc:407] Connection header received
0929 19:03:41.502293 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:41.502294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:41.502308 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:41.502422 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:41.502856 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:41.503449 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:41.504211 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:41.504419 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:41.504473 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:41.507112 (+ 2639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.65:55347: BlockingRecv error: recv got EOF from 127.8.158.65:55347 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":61,"threads_started":1}
Sep 29 19:03:41 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.66@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:41 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.66@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:41.638135 18001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:41.632485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.66:56775)
0929 19:03:41.632620 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:41.632632 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:41.632688 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:41.632854 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:41.632855 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:41.633020 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:41.633062 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:41.633064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:41.633758 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:41.633759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:41.634527 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:41.634529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:41.635011 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:41.635012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:41.635055 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:41.635566 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:41.635571 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:41.638057 (+ 2486us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
W20260929 19:03:41.638373 18000 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:41.632634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:56775 (local address 127.8.158.126:40017)
0929 19:03:41.632722 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:41.632724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:41.632733 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:03:41.632760 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:41.632761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:41.632772 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:41.632830 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:41.633157 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:41.633661 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:41.634681 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:41.634936 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:41.634995 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:41.638209 (+ 3214us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.66:56775: BlockingRecv error: recv got EOF from 127.8.158.66:56775 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 29 19:03:41 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.67@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:41 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.67@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:41.785454 18002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:41.780733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.67:41167)
0929 19:03:41.780926 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:41.780937 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:41.780998 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:41.781165 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:41.781166 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:41.781321 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:41.781366 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:41.781367 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:41.782075 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:41.782076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:41.782639 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:41.782641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:41.783078 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:41.783078 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:41.783123 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:41.783636 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:41.783641 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:41.785389 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":107,"threads_started":1}
W20260929 19:03:41.785549 18000 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:41.780818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:41167 (local address 127.8.158.126:40017)
0929 19:03:41.780897 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:41.780899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:41.781033 (+ 134us) server_negotiation.cc:407] Connection header received
0929 19:03:41.781057 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:41.781058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:41.781070 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:41.781126 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:41.781442 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:41.781996 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:41.782775 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:41.783010 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:41.783076 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:41.785478 (+ 2402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.67:41167: BlockingRecv error: recv got EOF from 127.8.158.67:41167 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 19:03:42 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.65@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:42 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.65@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:42.512446 18003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:42.507648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.65:50703)
0929 19:03:42.507787 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:42.507797 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:42.507845 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:42.508039 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:42.508039 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:42.508200 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:42.508243 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:42.508245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:42.508920 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:42.508921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:42.509513 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:42.509515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:42.509902 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:42.509904 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:42.509952 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:42.510476 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:42.510481 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:42.512388 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":76,"threads_started":1}
W20260929 19:03:42.512553 18004 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:42.507801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:50703 (local address 127.8.158.126:40017)
0929 19:03:42.507913 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:42.507915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:42.507930 (+ 15us) server_negotiation.cc:407] Connection header received
0929 19:03:42.507949 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:42.507950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:42.507960 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:42.508022 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:42.508307 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:42.508827 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:42.509603 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:42.509814 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:42.509872 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:42.512476 (+ 2604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.65:50703: BlockingRecv error: recv got EOF from 127.8.158.65:50703 (error 108)
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":65,"threads_started":1}
Sep 29 19:03:42 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.66@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:42 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.66@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:42.643610 18005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:42.638594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.66:33203)
0929 19:03:42.638737 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:42.638748 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:42.638812 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:42.638960 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:42.638960 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:42.639117 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:42.639161 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:42.639162 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:42.639875 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:42.639876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:42.640486 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:42.640488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:42.640907 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:42.640908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:42.640959 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:42.641455 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:42.641459 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:42.643494 (+ 2035us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":88,"threads_started":1}
W20260929 19:03:42.643647 18004 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:42.638697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:33203 (local address 127.8.158.126:40017)
0929 19:03:42.638769 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:42.638772 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:42.638798 (+ 26us) server_negotiation.cc:407] Connection header received
0929 19:03:42.638870 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:42.638870 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:42.638881 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:42.638948 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:42.639229 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:42.639760 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:42.640591 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:42.640811 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:42.640878 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:42.643583 (+ 2705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.66:33203: BlockingRecv error: recv got EOF from 127.8.158.66:33203 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 29 19:03:42 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.67@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:42 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.67@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:42.790817 18006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:42.785935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.67:38571)
0929 19:03:42.786095 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:42.786107 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:42.786199 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:42.786388 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:42.786389 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:42.786618 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:42.786672 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:42.786675 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:42.787467 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:42.787468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:42.788074 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:42.788076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:42.788425 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:42.788426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:42.788470 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:42.788948 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:42.788953 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:42.790732 (+ 1779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":114,"threads_started":1}
W20260929 19:03:42.791020 18004 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:42.786032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:38571 (local address 127.8.158.126:40017)
0929 19:03:42.786150 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:42.786152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:42.786196 (+ 44us) server_negotiation.cc:407] Connection header received
0929 19:03:42.786268 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:42.786269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:42.786280 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:42.786378 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:42.786771 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:42.787364 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:42.788172 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:42.788353 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:42.788409 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:42.790936 (+ 2527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.67:38571: BlockingRecv error: recv got EOF from 127.8.158.67:38571 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 29 19:03:43 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.65@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:43 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.65@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:43.517617 18007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:43.513016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.65:37507)
0929 19:03:43.513179 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:43.513190 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:43.513241 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:43.513390 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:43.513391 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:43.513553 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:43.513597 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:43.513598 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:43.514233 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:43.514234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:43.514808 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:43.514810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:43.515262 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:43.515263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:43.515328 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:43.515841 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:43.515853 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:43.517551 (+ 1698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
W20260929 19:03:43.517748 18008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:43.513088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:37507 (local address 127.8.158.126:40017)
0929 19:03:43.513247 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:43.513249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:43.513259 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:03:43.513296 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:43.513296 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:43.513306 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:43.513373 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:43.513663 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:43.514160 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:43.514889 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:43.515181 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:43.515262 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:43.517644 (+ 2382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.65:37507: BlockingRecv error: recv got EOF from 127.8.158.65:37507 (error 108)
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":94,"threads_started":1}
Sep 29 19:03:43 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.66@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:43 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.66@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:43.649086 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:43.644290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.66:44927)
0929 19:03:43.644435 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:43.644448 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:43.644516 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:43.644649 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:43.644650 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:43.644844 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:43.644918 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:43.644920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:43.645602 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:43.645603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:43.646135 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:43.646137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:43.646596 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:43.646597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:43.646661 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:43.647162 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:43.647167 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:43.648989 (+ 1822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":63,"threads_started":1}
W20260929 19:03:43.649129 18008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:43.644368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:44927 (local address 127.8.158.126:40017)
0929 19:03:43.644448 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:43.644450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:43.644498 (+ 48us) server_negotiation.cc:407] Connection header received
0929 19:03:43.644572 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:43.644573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:43.644584 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:43.644639 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:43.645013 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:43.645510 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:43.646234 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:43.646504 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:43.646579 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:43.649067 (+ 2488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.66:44927: BlockingRecv error: recv got EOF from 127.8.158.66:44927 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 29 19:03:43 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.67@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:43 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.67@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:43.796087 18010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:43.791314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.67:56243)
0929 19:03:43.791489 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:43.791499 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:43.791561 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:43.791691 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:43.791692 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:43.791856 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:43.791902 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:43.791904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:43.792545 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:43.792546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:43.793091 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:43.793093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:43.793451 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:43.793452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:43.793498 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:43.793957 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:43.793961 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:43.795978 (+ 2017us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":101,"threads_started":1}
W20260929 19:03:43.796139 18008 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:43.791377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:56243 (local address 127.8.158.126:40017)
0929 19:03:43.791444 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:43.791447 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:43.791557 (+ 110us) server_negotiation.cc:407] Connection header received
0929 19:03:43.791614 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:43.791614 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:43.791626 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:43.791666 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:43.791965 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:43.792456 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:43.793188 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:43.793372 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:43.793437 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:43.796054 (+ 2617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.67:56243: BlockingRecv error: recv got EOF from 127.8.158.67:56243 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 29 19:03:44 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.65@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:44 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.65@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:44.522903 18011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:44.518135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.65:60567)
0929 19:03:44.518285 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:44.518295 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:44.518365 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:44.518623 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:44.518625 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:44.518786 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:44.518832 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:44.518834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:44.519580 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:44.519581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:44.520089 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:44.520091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:44.520498 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:44.520500 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:44.520558 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:44.520986 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:44.520990 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:44.522832 (+ 1842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":88,"threads_started":1}
W20260929 19:03:44.523073 18012 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:44.518203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:60567 (local address 127.8.158.126:40017)
0929 19:03:44.518373 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:44.518375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:44.518406 (+ 31us) server_negotiation.cc:407] Connection header received
0929 19:03:44.518428 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:44.518428 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:44.518447 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:44.518498 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:44.518977 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:44.519494 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:44.520184 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:44.520427 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:44.520494 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:44.522950 (+ 2456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.65:60567: BlockingRecv error: recv got EOF from 127.8.158.65:60567 (error 108)
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":76,"threads_started":1}
Sep 29 19:03:44 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.66@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:44 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.66@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:44.654618 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:44.649644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.66:59971)
0929 19:03:44.649781 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:44.649795 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:44.649852 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:44.649988 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:44.649989 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:44.650261 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:44.650349 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:44.650351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:44.651168 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:44.651169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:44.651715 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:44.651717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:44.652077 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:44.652078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:44.652127 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:44.652602 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:44.652609 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:44.654547 (+ 1938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":90,"threads_started":1}
W20260929 19:03:44.654872 18012 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:44.649738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:59971 (local address 127.8.158.126:40017)
0929 19:03:44.649833 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:44.649835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:44.649847 (+ 12us) server_negotiation.cc:407] Connection header received
0929 19:03:44.649902 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:44.649903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:44.649913 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:44.649972 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:44.650460 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:44.651092 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:44.651802 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:44.651992 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:44.652049 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:44.654817 (+ 2768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.66:59971: BlockingRecv error: recv got EOF from 127.8.158.66:59971 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 29 19:03:44 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.67@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:44 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.67@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:44.801369 18014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:44.796637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.67:57695)
0929 19:03:44.796785 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:44.796802 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:44.796867 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:44.797007 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:44.797007 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:44.797181 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:44.797228 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:44.797229 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:44.797866 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:44.797867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:44.798482 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:44.798485 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:44.798868 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:44.798870 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:44.798917 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:44.799372 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:44.799376 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:44.801296 (+ 1920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":90,"threads_started":1}
W20260929 19:03:44.801544 18012 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:44.796705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:57695 (local address 127.8.158.126:40017)
0929 19:03:44.796772 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:44.796774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:44.796860 (+ 86us) server_negotiation.cc:407] Connection header received
0929 19:03:44.796920 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:44.796921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:44.796932 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:44.796984 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:44.797286 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:44.797783 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:44.798573 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:44.798788 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:44.798850 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:44.801398 (+ 2548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.67:57695: BlockingRecv error: recv got EOF from 127.8.158.67:57695 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 29 19:03:45 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.65@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:45 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.65@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:45.535560 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:45.523408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.65:51973)
0929 19:03:45.523554 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:45.523566 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:45.523665 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:45.523855 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:45.523856 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:45.524016 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:45.524061 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:45.524063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:45.524793 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:45.524794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:45.525336 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:45.525338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:45.525774 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:45.525775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:45.525825 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:45.533366 (+ 7541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:45.533372 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:45.535486 (+ 2114us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":86,"threads_started":1}
W20260929 19:03:45.535674 18016 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:45.523531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:51973 (local address 127.8.158.126:40017)
0929 19:03:45.523647 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:45.523649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:45.523661 (+ 12us) server_negotiation.cc:407] Connection header received
0929 19:03:45.523755 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:45.523756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:45.523766 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:45.523844 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:45.524150 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:45.524708 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:45.525418 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:45.525666 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:45.525727 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:45.535595 (+ 9868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.65:51973: BlockingRecv error: recv got EOF from 127.8.158.65:51973 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":57,"threads_started":1}
Sep 29 19:03:45 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.66@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:03:45 dist-test-slave-nmsz krb5kdc[16958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.158.66@KRBTEST.COM for oryx/127.8.158.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:03:45.660142 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:45.655184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:40017 (local address 127.8.158.66:47955)
0929 19:03:45.655360 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:45.655372 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:45.655444 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:45.655560 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:45.655561 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:45.655763 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:45.655811 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:45.655813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:45.656426 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:45.656427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:45.657015 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:45.657017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:45.657364 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:45.657365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:45.657418 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:45.658038 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:45.658043 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:45.660047 (+ 2004us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.158.126:40017: Server oryx/127.8.158.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":88,"threads_started":1}
W20260929 19:03:45.660190 18016 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:45.655294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:47955 (local address 127.8.158.126:40017)
0929 19:03:45.655366 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:45.655368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:45.655438 (+ 70us) server_negotiation.cc:407] Connection header received
0929 19:03:45.655498 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:45.655499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:45.655509 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:45.655554 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:45.655874 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:45.656358 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:45.657082 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:45.657264 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:45.657313 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:45.660121 (+ 2808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.158.66:47955: BlockingRecv error: recv got EOF from 127.8.158.66:47955 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 19:03:45.724067 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 17541
I20260929 19:03:45.728987 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 17693
I20260929 19:03:45.733711 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 17826
I20260929 19:03:45.738062 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 17471
2026-09-29T19:03:45Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17559 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T19:03:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:03:45Z Disabled control of system clock
I20260929 19:03:45.765821 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:36037
--webserver_interface=127.8.158.126
--webserver_port=0
--builtin_ntp_servers=127.8.158.84:46061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:36037
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 19:03:45.843567 18021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:45.843755 18021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:45.843775 18021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:45.844853 18021 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:03:45.844887 18021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:45.844900 18021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:45.844913 18021 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:03:45.844925 18021 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:03:45.846170 18021 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:46061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:36037
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:36037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.158.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:45.846431 18021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:45.846681 18021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:45.851271 18026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:45.851263 18027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:45.851263 18029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:45.851802 18021 server_base.cc:1034] running on GCE node
I20260929 19:03:45.852073 18021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:45.852360 18021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:45.853520 18021 hybrid_clock.cc:648] HybridClock initialized: now 1790708625853498 us; error 37 us; skew 500 ppm
I20260929 19:03:45.854749 18021 webserver.cc:460] Webserver started at http://127.8.158.126:37319/ using document root <none> and password file <none>
I20260929 19:03:45.854995 18021 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:45.855063 18021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:45.855224 18021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:45.856328 18021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "e02e6a94faac4e55a414b80abf09ec98"
format_stamp: "Formatted at 2026-09-29 19:03:45 on dist-test-slave-nmsz"
server_key: "cabb60d7ab83373ec6eb3b9775a7ead0"
server_key_iv: "787533a620564236608b61119530b511"
server_key_version: "encryptionkey@0"
I20260929 19:03:45.856678 18021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "e02e6a94faac4e55a414b80abf09ec98"
format_stamp: "Formatted at 2026-09-29 19:03:45 on dist-test-slave-nmsz"
server_key: "cabb60d7ab83373ec6eb3b9775a7ead0"
server_key_iv: "787533a620564236608b61119530b511"
server_key_version: "encryptionkey@0"
I20260929 19:03:45.857939 18021 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:03:45.858670 18035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:45.858850 18021 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:45.858947 18021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "e02e6a94faac4e55a414b80abf09ec98"
format_stamp: "Formatted at 2026-09-29 19:03:45 on dist-test-slave-nmsz"
server_key: "cabb60d7ab83373ec6eb3b9775a7ead0"
server_key_iv: "787533a620564236608b61119530b511"
server_key_version: "encryptionkey@0"
I20260929 19:03:45.859012 18021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:45.883517 18021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:45.883783 18021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:45.883893 18021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:45.887884 18021 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.126:36037
I20260929 19:03:45.887916 18087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.126:36037 every 8 connection(s)
I20260929 19:03:45.888255 18021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:03:45.888913 18088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:45.889899 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 18021
I20260929 19:03:45.889972 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:03:45.890285 8825 external_mini_cluster.cc:1428] Setting key e0914afd81a91d14ecc111bd5f8dc0fa
I20260929 19:03:45.892414 18088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e02e6a94faac4e55a414b80abf09ec98: Bootstrap starting.
I20260929 19:03:45.893086 18088 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e02e6a94faac4e55a414b80abf09ec98: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:45.893359 18088 log.cc:824] T 00000000000000000000000000000000 P e02e6a94faac4e55a414b80abf09ec98: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:45.894099 18088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e02e6a94faac4e55a414b80abf09ec98: No bootstrap required, opened a new log
I20260929 19:03:45.895815 18088 raft_consensus.cc:348] T 00000000000000000000000000000000 P e02e6a94faac4e55a414b80abf09ec98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e02e6a94faac4e55a414b80abf09ec98" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 36037 } }
I20260929 19:03:45.895979 18088 raft_consensus.cc:374] T 00000000000000000000000000000000 P e02e6a94faac4e55a414b80abf09ec98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:45.896023 18088 raft_consensus.cc:729] T 00000000000000000000000000000000 P e02e6a94faac4e55a414b80abf09ec98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e02e6a94faac4e55a414b80abf09ec98, State: Initialized, Role: FOLLOWER
I20260929 19:03:45.896157 18088 consensus_queue.cc:260] T 00000000000000000000000000000000 P e02e6a94faac4e55a414b80abf09ec98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e02e6a94faac4e55a414b80abf09ec98" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 36037 } }
I20260929 19:03:45.896256 18088 raft_consensus.cc:388] T 00000000000000000000000000000000 P e02e6a94faac4e55a414b80abf09ec98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:03:45.896292 18088 raft_consensus.cc:482] T 00000000000000000000000000000000 P e02e6a94faac4e55a414b80abf09ec98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:03:45.896346 18088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e02e6a94faac4e55a414b80abf09ec98 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:45.896445 18091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:45.890858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49758 (local address 127.8.158.126:36037)
0929 19:03:45.891059 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:45.891068 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:45.891099 (+ 31us) server_negotiation.cc:407] Connection header received
0929 19:03:45.891598 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:45.891606 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:45.891685 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:45.891829 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:45.892031 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:45.893330 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:45.894098 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:45.894700 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:45.894825 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:45.895474 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:45.895478 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:45.895478 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:03:45.895679 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:45.895718 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:03:45.895732 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:45.895801 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:45.895869 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 19:03:45.895940 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":76,"threads_started":1}
I20260929 19:03:45.896944 18088 raft_consensus.cc:504] T 00000000000000000000000000000000 P e02e6a94faac4e55a414b80abf09ec98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e02e6a94faac4e55a414b80abf09ec98" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 36037 } }
I20260929 19:03:45.897060 18088 leader_election.cc:302] T 00000000000000000000000000000000 P e02e6a94faac4e55a414b80abf09ec98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e02e6a94faac4e55a414b80abf09ec98; no voters:
I20260929 19:03:45.897212 18088 leader_election.cc:288] T 00000000000000000000000000000000 P e02e6a94faac4e55a414b80abf09ec98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:03:45.897258 18093 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e02e6a94faac4e55a414b80abf09ec98 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:03:45.897406 18093 raft_consensus.cc:686] T 00000000000000000000000000000000 P e02e6a94faac4e55a414b80abf09ec98 [term 1 LEADER]: Becoming Leader. State: Replica: e02e6a94faac4e55a414b80abf09ec98, State: Running, Role: LEADER
I20260929 19:03:45.897505 18088 sys_catalog.cc:564] T 00000000000000000000000000000000 P e02e6a94faac4e55a414b80abf09ec98 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:03:45.897531 18093 consensus_queue.cc:237] T 00000000000000000000000000000000 P e02e6a94faac4e55a414b80abf09ec98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e02e6a94faac4e55a414b80abf09ec98" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 36037 } }
I20260929 19:03:45.897943 18094 sys_catalog.cc:455] T 00000000000000000000000000000000 P e02e6a94faac4e55a414b80abf09ec98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e02e6a94faac4e55a414b80abf09ec98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e02e6a94faac4e55a414b80abf09ec98" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 36037 } } }
I20260929 19:03:45.897954 18095 sys_catalog.cc:455] T 00000000000000000000000000000000 P e02e6a94faac4e55a414b80abf09ec98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e02e6a94faac4e55a414b80abf09ec98. Latest consensus state: current_term: 1 leader_uuid: "e02e6a94faac4e55a414b80abf09ec98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e02e6a94faac4e55a414b80abf09ec98" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 36037 } } }
I20260929 19:03:45.898138 18095 sys_catalog.cc:458] T 00000000000000000000000000000000 P e02e6a94faac4e55a414b80abf09ec98 [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:45.898126 18094 sys_catalog.cc:458] T 00000000000000000000000000000000 P e02e6a94faac4e55a414b80abf09ec98 [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:45.898566 18102 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:03:45.899021 18102 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:03:45.900453 18102 catalog_manager.cc:1347] Generated new cluster ID: 1f9016e99f56481ea7a1958bc19bd20a
I20260929 19:03:45.900502 18102 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:03:45.915169 18102 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:03:45.915854 18102 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:03:45.921223 18102 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e02e6a94faac4e55a414b80abf09ec98: Generated new TSK 0
I20260929 19:03:45.921391 18102 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:03:45.923956 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.65:0
--local_ip_for_outbound_sockets=127.8.158.65
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:36037
--builtin_ntp_servers=127.8.158.84:46061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 19:03:46.007601 18112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:46.007942 18112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:46.008023 18112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:46.009864 18112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:46.009922 18112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:46.009979 18112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.65
I20260929 19:03:46.011423 18112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:46061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:36037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.65
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:46.011672 18112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:46.011984 18112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:46.016952 18118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:46.016988 18117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:46.016958 18120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:46.017216 18112 server_base.cc:1034] running on GCE node
I20260929 19:03:46.017447 18112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:46.017685 18112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:46.018838 18112 hybrid_clock.cc:648] HybridClock initialized: now 1790708626018836 us; error 43 us; skew 500 ppm
I20260929 19:03:46.020154 18112 webserver.cc:460] Webserver started at http://127.8.158.65:46309/ using document root <none> and password file <none>
I20260929 19:03:46.020413 18112 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:46.020467 18112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:46.020587 18112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:46.021884 18112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/data/instance:
uuid: "534956abbe454a2582be0e531b17a2cc"
format_stamp: "Formatted at 2026-09-29 19:03:46 on dist-test-slave-nmsz"
server_key: "0fc0a069bdd3845fecc49e74f293855d"
server_key_iv: "ce10e76732906dcd6307b2723de1cf9a"
server_key_version: "encryptionkey@0"
I20260929 19:03:46.022291 18112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance:
uuid: "534956abbe454a2582be0e531b17a2cc"
format_stamp: "Formatted at 2026-09-29 19:03:46 on dist-test-slave-nmsz"
server_key: "0fc0a069bdd3845fecc49e74f293855d"
server_key_iv: "ce10e76732906dcd6307b2723de1cf9a"
server_key_version: "encryptionkey@0"
I20260929 19:03:46.023923 18112 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:46.024770 18126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:46.025002 18112 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:46.025084 18112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/wal
uuid: "534956abbe454a2582be0e531b17a2cc"
format_stamp: "Formatted at 2026-09-29 19:03:46 on dist-test-slave-nmsz"
server_key: "0fc0a069bdd3845fecc49e74f293855d"
server_key_iv: "ce10e76732906dcd6307b2723de1cf9a"
server_key_version: "encryptionkey@0"
I20260929 19:03:46.025147 18112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:46.037488 18112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:46.037700 18112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:46.037793 18112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:46.037942 18112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:46.038242 18112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:46.038281 18112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:46.038385 18112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:46.038414 18112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:46.043924 18112 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.65:36549
I20260929 19:03:46.043960 18239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.65:36549 every 8 connection(s)
I20260929 19:03:46.044262 18112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
I20260929 19:03:46.049504 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 18112
I20260929 19:03:46.049593 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance
I20260929 19:03:46.049708 18242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:46.044961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:36037 (local address 127.8.158.65:41165)
0929 19:03:46.045150 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:46.045185 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:46.045676 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:46.045928 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:46.045932 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:46.045944 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:46.046493 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:46.046498 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:46.047202 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:46.047203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:46.047974 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:46.047976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:46.048374 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:46.048375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:46.048506 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:46.049016 (+ 510us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:03:46.049017 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:46.049043 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:03:46.049063 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:46.049271 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:46.049277 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:46.049280 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:03:46.049353 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 19:03:46.049464 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":79,"threads_started":1}
I20260929 19:03:46.049827 8825 external_mini_cluster.cc:1428] Setting key 25ea8a4397f9ae75c6eeb45ed8b9af77
I20260929 19:03:46.050071 18091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:46.044977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:41165 (local address 127.8.158.126:36037)
0929 19:03:46.045018 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:46.045020 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:46.045264 (+ 244us) server_negotiation.cc:407] Connection header received
0929 19:03:46.045789 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:46.045791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:46.045802 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:46.045860 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:46.046593 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:46.047093 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:46.048081 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:46.048286 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:46.048367 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:46.049171 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:46.049172 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:46.049173 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:03:46.049178 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:46.049192 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:03:46.049199 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:46.049269 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:46.049974 (+ 705us) server_negotiation.cc:299] Negotiation successful
0929 19:03:46.049999 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 19:03:46.050478 18240 heartbeater.cc:347] Connected to a master server at 127.8.158.126:36037
I20260929 19:03:46.050591 18240 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:46.050765 18240 heartbeater.cc:511] Master 127.8.158.126:36037 requested a full tablet report, sending...
I20260929 19:03:46.050906 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.66:0
--local_ip_for_outbound_sockets=127.8.158.66
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:36037
--builtin_ntp_servers=127.8.158.84:46061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 19:03:46.051285 18052 ts_manager.cc:194] Registered new tserver with Master: 534956abbe454a2582be0e531b17a2cc (127.8.158.65:36549)
I20260929 19:03:46.052174 18052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.158.65:41165
W20260929 19:03:46.131338 18243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:46.131531 18243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:46.131551 18243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:46.132720 18243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:46.132758 18243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:46.132783 18243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.66
I20260929 19:03:46.134135 18243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:46061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:36037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.66
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:46.134423 18243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:46.134748 18243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:46.139665 18251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:46.139690 18249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:46.139683 18248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:46.140048 18243 server_base.cc:1034] running on GCE node
I20260929 19:03:46.140260 18243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:46.140452 18243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:46.141634 18243 hybrid_clock.cc:648] HybridClock initialized: now 1790708626141617 us; error 26 us; skew 500 ppm
I20260929 19:03:46.143066 18243 webserver.cc:460] Webserver started at http://127.8.158.66:40403/ using document root <none> and password file <none>
I20260929 19:03:46.143333 18243 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:46.143404 18243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:46.143549 18243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:46.144560 18243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/data/instance:
uuid: "7b06280072da4663916b1711a6d6470a"
format_stamp: "Formatted at 2026-09-29 19:03:46 on dist-test-slave-nmsz"
server_key: "894999938b6bc93c27d982cae274fe63"
server_key_iv: "6da69e1b928ab36ec2c1452b64c350d5"
server_key_version: "encryptionkey@0"
I20260929 19:03:46.144919 18243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance:
uuid: "7b06280072da4663916b1711a6d6470a"
format_stamp: "Formatted at 2026-09-29 19:03:46 on dist-test-slave-nmsz"
server_key: "894999938b6bc93c27d982cae274fe63"
server_key_iv: "6da69e1b928ab36ec2c1452b64c350d5"
server_key_version: "encryptionkey@0"
I20260929 19:03:46.146265 18243 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:03:46.147063 18257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:46.147219 18243 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:03:46.147315 18243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/wal
uuid: "7b06280072da4663916b1711a6d6470a"
format_stamp: "Formatted at 2026-09-29 19:03:46 on dist-test-slave-nmsz"
server_key: "894999938b6bc93c27d982cae274fe63"
server_key_iv: "6da69e1b928ab36ec2c1452b64c350d5"
server_key_version: "encryptionkey@0"
I20260929 19:03:46.147389 18243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:46.156623 18243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:46.156881 18243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:46.157021 18243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:46.157234 18243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:46.157524 18243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:46.157577 18243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:46.157616 18243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:46.157642 18243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:46.163347 18243 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.66:38873
I20260929 19:03:46.163393 18370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.66:38873 every 8 connection(s)
I20260929 19:03:46.163688 18243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
I20260929 19:03:46.165998 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 18243
I20260929 19:03:46.166064 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance
I20260929 19:03:46.166263 8825 external_mini_cluster.cc:1428] Setting key a363b3b9a141e3160df3a8e0c85ed449
I20260929 19:03:46.167238 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.67:0
--local_ip_for_outbound_sockets=127.8.158.67
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:36037
--builtin_ntp_servers=127.8.158.84:46061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 19:03:46.169085 18091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:46.164285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:58963 (local address 127.8.158.126:36037)
0929 19:03:46.164325 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:46.164327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:46.164613 (+ 286us) server_negotiation.cc:407] Connection header received
0929 19:03:46.165129 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:46.165130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:46.165141 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:46.165248 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:46.165911 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:46.166604 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:46.167663 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:46.167877 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:46.167955 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:46.168838 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:46.168839 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:46.168840 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:03:46.168845 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:46.168859 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:03:46.168863 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:46.168905 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:46.168980 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 19:03:46.169009 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 19:03:46.169324 18373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:46.164282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:36037 (local address 127.8.158.66:58963)
0929 19:03:46.164486 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:46.164525 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:46.165022 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:46.165319 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:46.165323 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:46.165333 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:46.165816 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:46.165820 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:46.166706 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:46.166707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:46.167565 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:46.167567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:46.167961 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:46.167962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:46.168101 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:46.168680 (+ 579us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:03:46.168680 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:46.168712 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:03:46.168734 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:46.168916 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:46.168921 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:46.168924 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:03:46.168985 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 19:03:46.169082 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":84,"threads_started":1}
I20260929 19:03:46.169728 18371 heartbeater.cc:347] Connected to a master server at 127.8.158.126:36037
I20260929 19:03:46.169813 18371 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:46.170006 18371 heartbeater.cc:511] Master 127.8.158.126:36037 requested a full tablet report, sending...
I20260929 19:03:46.170377 18052 ts_manager.cc:194] Registered new tserver with Master: 7b06280072da4663916b1711a6d6470a (127.8.158.66:38873)
I20260929 19:03:46.170750 18052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.158.66:58963
W20260929 19:03:46.245769 18374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:46.245997 18374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:46.246031 18374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:46.247208 18374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:46.247280 18374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:46.247323 18374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.67
I20260929 19:03:46.248821 18374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:46061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:36037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.67
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:46.249099 18374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:46.249491 18374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:46.254670 18382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:46.254691 18379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:46.254765 18374 server_base.cc:1034] running on GCE node
W20260929 19:03:46.254695 18380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:46.255110 18374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:46.255371 18374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:46.256546 18374 hybrid_clock.cc:648] HybridClock initialized: now 1790708626256522 us; error 55 us; skew 500 ppm
I20260929 19:03:46.257789 18374 webserver.cc:460] Webserver started at http://127.8.158.67:44305/ using document root <none> and password file <none>
I20260929 19:03:46.258042 18374 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:46.258109 18374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:46.258255 18374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:46.259305 18374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/data/instance:
uuid: "3157f6155446444997c77052bacc0b7e"
format_stamp: "Formatted at 2026-09-29 19:03:46 on dist-test-slave-nmsz"
server_key: "b11efdb924bcad891820dafe847b24c8"
server_key_iv: "e2f840feb913b11c31e2a74f1af29163"
server_key_version: "encryptionkey@0"
I20260929 19:03:46.259656 18374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance:
uuid: "3157f6155446444997c77052bacc0b7e"
format_stamp: "Formatted at 2026-09-29 19:03:46 on dist-test-slave-nmsz"
server_key: "b11efdb924bcad891820dafe847b24c8"
server_key_iv: "e2f840feb913b11c31e2a74f1af29163"
server_key_version: "encryptionkey@0"
I20260929 19:03:46.260903 18374 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:03:46.261628 18388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:46.261850 18374 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:03:46.261930 18374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/wal
uuid: "3157f6155446444997c77052bacc0b7e"
format_stamp: "Formatted at 2026-09-29 19:03:46 on dist-test-slave-nmsz"
server_key: "b11efdb924bcad891820dafe847b24c8"
server_key_iv: "e2f840feb913b11c31e2a74f1af29163"
server_key_version: "encryptionkey@0"
I20260929 19:03:46.262012 18374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:46.271220 18374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:46.271457 18374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:46.271580 18374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:46.271790 18374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:46.272030 18374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:46.272060 18374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:46.272079 18374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:46.272092 18374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:46.277675 18374 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.67:37019
I20260929 19:03:46.277729 18501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.67:37019 every 8 connection(s)
I20260929 19:03:46.278100 18374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
I20260929 19:03:46.281960 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 18374
I20260929 19:03:46.282045 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance
I20260929 19:03:46.282251 8825 external_mini_cluster.cc:1428] Setting key 9b34d7930e9687a3320af0d4ae510ee2
I20260929 19:03:46.283006 18091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:46.278790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:41835 (local address 127.8.158.126:36037)
0929 19:03:46.278833 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:46.278835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:46.279050 (+ 215us) server_negotiation.cc:407] Connection header received
0929 19:03:46.279476 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:46.279477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:46.279486 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:46.279538 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:46.280118 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:46.280583 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:46.281493 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:46.281671 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:46.281746 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:46.282743 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:46.282744 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:46.282744 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:03:46.282749 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:46.282762 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:03:46.282766 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:46.282832 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:46.282907 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 19:03:46.282932 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 19:03:46.283233 18504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:46.278763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:36037 (local address 127.8.158.67:41835)
0929 19:03:46.278934 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:46.278963 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:46.279392 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:46.279584 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:46.279587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:46.279594 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:46.280049 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:46.280052 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:46.280681 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:46.280682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:46.281414 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:46.281415 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:46.281764 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:46.281785 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:46.281944 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:46.282573 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:03:46.282574 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:46.282610 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:03:46.282636 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:46.282834 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:46.282840 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:46.282843 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:03:46.282912 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 19:03:46.283002 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260929 19:03:46.283715 18502 heartbeater.cc:347] Connected to a master server at 127.8.158.126:36037
I20260929 19:03:46.283787 18502 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:46.283974 18502 heartbeater.cc:511] Master 127.8.158.126:36037 requested a full tablet report, sending...
I20260929 19:03:46.284310 18052 ts_manager.cc:194] Registered new tserver with Master: 3157f6155446444997c77052bacc0b7e (127.8.158.67:37019)
I20260929 19:03:46.284633 18052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.158.67:41835
I20260929 19:03:46.293931 8825 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 19:03:46.295851 18091 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:46.295353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49772 (local address 127.8.158.126:36037)
0929 19:03:46.295413 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:46.295415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:46.295535 (+ 120us) server_negotiation.cc:407] Connection header received
0929 19:03:46.295587 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:46.295588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:46.295596 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:46.295652 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:46.295766 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49772: BlockingRecv error: recv got EOF from 127.0.0.1:49772 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 19:03:46.295912 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 18112
I20260929 19:03:46.300424 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 18243
I20260929 19:03:46.304917 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 18374
I20260929 19:03:46.308848 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 18021
2026-09-29T19:03:46Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (569 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T19:03:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:03:46Z Disabled control of system clock
I20260929 19:03:46.334692 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:44329
--webserver_interface=127.8.158.126
--webserver_port=0
--builtin_ntp_servers=127.8.158.84:38625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:44329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 19:03:46.414692 18513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:46.414889 18513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:46.414913 18513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:46.415993 18513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:03:46.416028 18513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:46.416041 18513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:46.416054 18513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:03:46.416066 18513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:03:46.417306 18513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:38625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:44329
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:44329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.158.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:46.417490 18513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:46.417703 18513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:46.422380 18521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:46.422380 18518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:46.422446 18519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:46.422571 18513 server_base.cc:1034] running on GCE node
I20260929 19:03:46.422919 18513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:46.423146 18513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:46.424288 18513 hybrid_clock.cc:648] HybridClock initialized: now 1790708626424276 us; error 27 us; skew 500 ppm
I20260929 19:03:46.425514 18513 webserver.cc:460] Webserver started at http://127.8.158.126:45277/ using document root <none> and password file <none>
I20260929 19:03:46.425753 18513 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:46.425824 18513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:46.425973 18513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:46.427074 18513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "27c0573e36544aa8b6376d0e688283e4"
format_stamp: "Formatted at 2026-09-29 19:03:46 on dist-test-slave-nmsz"
server_key: "83e65b8d16d4681d9d8fa2c8ae9af8ce"
server_key_iv: "804c0ad1e04f17c7a4fa7f14b095d5c5"
server_key_version: "encryptionkey@0"
I20260929 19:03:46.427429 18513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "27c0573e36544aa8b6376d0e688283e4"
format_stamp: "Formatted at 2026-09-29 19:03:46 on dist-test-slave-nmsz"
server_key: "83e65b8d16d4681d9d8fa2c8ae9af8ce"
server_key_iv: "804c0ad1e04f17c7a4fa7f14b095d5c5"
server_key_version: "encryptionkey@0"
I20260929 19:03:46.428735 18513 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:03:46.429411 18527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:46.429574 18513 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:03:46.429669 18513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "27c0573e36544aa8b6376d0e688283e4"
format_stamp: "Formatted at 2026-09-29 19:03:46 on dist-test-slave-nmsz"
server_key: "83e65b8d16d4681d9d8fa2c8ae9af8ce"
server_key_iv: "804c0ad1e04f17c7a4fa7f14b095d5c5"
server_key_version: "encryptionkey@0"
I20260929 19:03:46.429747 18513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:46.432946 18513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:46.433176 18513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:46.433324 18513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:46.437505 18513 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.126:44329
I20260929 19:03:46.437546 18579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.126:44329 every 8 connection(s)
I20260929 19:03:46.437842 18513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:03:46.438494 18580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:46.438496 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 18513
I20260929 19:03:46.438585 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:03:46.438786 8825 external_mini_cluster.cc:1428] Setting key a9cc71a73cfe4237b7a588e284b0d2e4
I20260929 19:03:46.441308 18583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:46.439338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44644 (local address 127.8.158.126:44329)
0929 19:03:46.439514 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:46.439523 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:46.439557 (+ 34us) server_negotiation.cc:407] Connection header received
0929 19:03:46.440083 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:46.440090 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:46.440140 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:46.440296 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:46.440904 (+ 608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:46.440907 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:46.440908 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:03:46.440912 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:46.440954 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:03:46.440967 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:46.441045 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:46.441083 (+ 38us) server_negotiation.cc:299] Negotiation successful
0929 19:03:46.441134 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":59,"threads_started":1}
I20260929 19:03:46.442081 18580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27c0573e36544aa8b6376d0e688283e4: Bootstrap starting.
I20260929 19:03:46.442679 18580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27c0573e36544aa8b6376d0e688283e4: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:46.442961 18580 log.cc:824] T 00000000000000000000000000000000 P 27c0573e36544aa8b6376d0e688283e4: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:46.443589 18580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27c0573e36544aa8b6376d0e688283e4: No bootstrap required, opened a new log
I20260929 19:03:46.445039 18580 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27c0573e36544aa8b6376d0e688283e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27c0573e36544aa8b6376d0e688283e4" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 44329 } }
I20260929 19:03:46.445158 18580 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27c0573e36544aa8b6376d0e688283e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:46.445180 18580 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27c0573e36544aa8b6376d0e688283e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27c0573e36544aa8b6376d0e688283e4, State: Initialized, Role: FOLLOWER
I20260929 19:03:46.445314 18580 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27c0573e36544aa8b6376d0e688283e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27c0573e36544aa8b6376d0e688283e4" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 44329 } }
I20260929 19:03:46.445397 18580 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27c0573e36544aa8b6376d0e688283e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:03:46.445442 18580 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27c0573e36544aa8b6376d0e688283e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:03:46.445488 18580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27c0573e36544aa8b6376d0e688283e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:46.446010 18580 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27c0573e36544aa8b6376d0e688283e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27c0573e36544aa8b6376d0e688283e4" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 44329 } }
I20260929 19:03:46.446091 18580 leader_election.cc:302] T 00000000000000000000000000000000 P 27c0573e36544aa8b6376d0e688283e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27c0573e36544aa8b6376d0e688283e4; no voters:
I20260929 19:03:46.446254 18580 leader_election.cc:288] T 00000000000000000000000000000000 P 27c0573e36544aa8b6376d0e688283e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:03:46.446547 18580 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27c0573e36544aa8b6376d0e688283e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:03:46.446638 18585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27c0573e36544aa8b6376d0e688283e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:03:46.446779 18585 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27c0573e36544aa8b6376d0e688283e4 [term 1 LEADER]: Becoming Leader. State: Replica: 27c0573e36544aa8b6376d0e688283e4, State: Running, Role: LEADER
I20260929 19:03:46.446918 18585 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27c0573e36544aa8b6376d0e688283e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27c0573e36544aa8b6376d0e688283e4" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 44329 } }
I20260929 19:03:46.447314 18586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27c0573e36544aa8b6376d0e688283e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27c0573e36544aa8b6376d0e688283e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27c0573e36544aa8b6376d0e688283e4" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 44329 } } }
I20260929 19:03:46.447350 18587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27c0573e36544aa8b6376d0e688283e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27c0573e36544aa8b6376d0e688283e4. Latest consensus state: current_term: 1 leader_uuid: "27c0573e36544aa8b6376d0e688283e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27c0573e36544aa8b6376d0e688283e4" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 44329 } } }
I20260929 19:03:46.447481 18586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27c0573e36544aa8b6376d0e688283e4 [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:46.447515 18587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27c0573e36544aa8b6376d0e688283e4 [sys.catalog]: This master's current role is: LEADER
W20260929 19:03:46.448320 18601 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 27c0573e36544aa8b6376d0e688283e4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 19:03:46.448382 18601 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:03:46.448429 18596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:03:46.448571 18596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:03:46.449905 18596 catalog_manager.cc:1347] Generated new cluster ID: c31a335fc5294801b11f2452ea4a2f3a
I20260929 19:03:46.449955 18596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:03:46.456023 18596 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:03:46.456167 18596 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:03:46.462498 18596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27c0573e36544aa8b6376d0e688283e4: Generated new TSK 0
I20260929 19:03:46.462659 18596 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:03:46.465142 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.65:0
--local_ip_for_outbound_sockets=127.8.158.65
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:44329
--builtin_ntp_servers=127.8.158.84:38625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 19:03:46.543658 18604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:46.543843 18604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:46.543864 18604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:46.544991 18604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:46.545028 18604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:46.545053 18604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.65
I20260929 19:03:46.546428 18604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:38625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:44329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.65
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:46.546654 18604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:46.546947 18604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:46.552687 18609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:46.552726 18610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:46.552937 18612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:46.552946 18604 server_base.cc:1034] running on GCE node
I20260929 19:03:46.553164 18604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:46.553377 18604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:46.554538 18604 hybrid_clock.cc:648] HybridClock initialized: now 1790708626554525 us; error 29 us; skew 500 ppm
I20260929 19:03:46.555749 18604 webserver.cc:460] Webserver started at http://127.8.158.65:45243/ using document root <none> and password file <none>
I20260929 19:03:46.556010 18604 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:46.556082 18604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:46.556231 18604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:46.557248 18604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/data/instance:
uuid: "7b9d336af68845a5884894da954a3ebe"
format_stamp: "Formatted at 2026-09-29 19:03:46 on dist-test-slave-nmsz"
server_key: "38f142bb64fd387d67f906b0b6fa0971"
server_key_iv: "eb3acf2414e7f5ac9802897c5be370c5"
server_key_version: "encryptionkey@0"
I20260929 19:03:46.557595 18604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance:
uuid: "7b9d336af68845a5884894da954a3ebe"
format_stamp: "Formatted at 2026-09-29 19:03:46 on dist-test-slave-nmsz"
server_key: "38f142bb64fd387d67f906b0b6fa0971"
server_key_iv: "eb3acf2414e7f5ac9802897c5be370c5"
server_key_version: "encryptionkey@0"
I20260929 19:03:46.558986 18604 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:03:46.559780 18618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:46.559935 18604 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:03:46.560010 18604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/wal
uuid: "7b9d336af68845a5884894da954a3ebe"
format_stamp: "Formatted at 2026-09-29 19:03:46 on dist-test-slave-nmsz"
server_key: "38f142bb64fd387d67f906b0b6fa0971"
server_key_iv: "eb3acf2414e7f5ac9802897c5be370c5"
server_key_version: "encryptionkey@0"
I20260929 19:03:46.560099 18604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:46.563314 18604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:46.563567 18604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:46.563699 18604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:46.563916 18604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:46.564205 18604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:46.564257 18604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:46.564287 18604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:46.564316 18604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:46.570525 18604 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.65:41267
I20260929 19:03:46.570567 18731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.65:41267 every 8 connection(s)
I20260929 19:03:46.570943 18604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
I20260929 19:03:46.573838 18583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:46.571673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:43169 (local address 127.8.158.126:44329)
0929 19:03:46.571730 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:46.571732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:46.571963 (+ 231us) server_negotiation.cc:407] Connection header received
0929 19:03:46.572410 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:46.572411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:46.572417 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:46.572501 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:46.573650 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:46.573651 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:46.573651 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:03:46.573654 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:46.573665 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:03:46.573669 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:46.573715 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:46.573777 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 19:03:46.573787 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 19:03:46.574049 18734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:46.571657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:44329 (local address 127.8.158.65:43169)
0929 19:03:46.571831 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:46.571871 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:46.572314 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:46.572550 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:46.572553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:46.572561 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:46.573511 (+ 950us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:03:46.573512 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:46.573539 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:03:46.573567 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:46.573716 (+ 149us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:46.573718 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:46.573720 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 19:03:46.573780 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 19:03:46.573824 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
I20260929 19:03:46.574532 18732 heartbeater.cc:347] Connected to a master server at 127.8.158.126:44329
I20260929 19:03:46.574604 18732 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:46.574708 18732 heartbeater.cc:511] Master 127.8.158.126:44329 requested a full tablet report, sending...
I20260929 19:03:46.575160 18544 ts_manager.cc:194] Registered new tserver with Master: 7b9d336af68845a5884894da954a3ebe (127.8.158.65:41267)
I20260929 19:03:46.579638 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 18604
I20260929 19:03:46.579735 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance
I20260929 19:03:46.579931 8825 external_mini_cluster.cc:1428] Setting key 12db68914ed712574dd32c9a9cd0235b
I20260929 19:03:46.580770 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.66:0
--local_ip_for_outbound_sockets=127.8.158.66
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:44329
--builtin_ntp_servers=127.8.158.84:38625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 19:03:46.659714 18735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:46.659941 18735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:46.659977 18735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:46.661123 18735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:46.661188 18735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:46.661232 18735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.66
I20260929 19:03:46.662698 18735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:38625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:44329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.66
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:46.662949 18735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:46.663228 18735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:46.668246 18740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:46.668279 18741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:46.668309 18735 server_base.cc:1034] running on GCE node
W20260929 19:03:46.668274 18743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:46.668716 18735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:46.668954 18735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:46.670120 18735 hybrid_clock.cc:648] HybridClock initialized: now 1790708626670099 us; error 34 us; skew 500 ppm
I20260929 19:03:46.671460 18735 webserver.cc:460] Webserver started at http://127.8.158.66:44429/ using document root <none> and password file <none>
I20260929 19:03:46.671715 18735 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:46.671804 18735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:46.671942 18735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:46.672946 18735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/data/instance:
uuid: "0519336f5e2d459ab344adc10f6c1f60"
format_stamp: "Formatted at 2026-09-29 19:03:46 on dist-test-slave-nmsz"
server_key: "b994cab60ca89396ba30f723fd13e5a7"
server_key_iv: "9c057ff79ccd43bd22db69f47bfc7199"
server_key_version: "encryptionkey@0"
I20260929 19:03:46.673308 18735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance:
uuid: "0519336f5e2d459ab344adc10f6c1f60"
format_stamp: "Formatted at 2026-09-29 19:03:46 on dist-test-slave-nmsz"
server_key: "b994cab60ca89396ba30f723fd13e5a7"
server_key_iv: "9c057ff79ccd43bd22db69f47bfc7199"
server_key_version: "encryptionkey@0"
I20260929 19:03:46.674587 18735 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:03:46.675278 18749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:46.675448 18735 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:03:46.675542 18735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/wal
uuid: "0519336f5e2d459ab344adc10f6c1f60"
format_stamp: "Formatted at 2026-09-29 19:03:46 on dist-test-slave-nmsz"
server_key: "b994cab60ca89396ba30f723fd13e5a7"
server_key_iv: "9c057ff79ccd43bd22db69f47bfc7199"
server_key_version: "encryptionkey@0"
I20260929 19:03:46.675622 18735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:46.678843 18735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:46.679090 18735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:46.679226 18735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:46.679437 18735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:46.679718 18735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:46.679774 18735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:46.679813 18735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:46.679842 18735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:46.685590 18735 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.66:44757
I20260929 19:03:46.685629 18862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.66:44757 every 8 connection(s)
I20260929 19:03:46.685941 18735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
I20260929 19:03:46.688786 18583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:46.686641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:52933 (local address 127.8.158.126:44329)
0929 19:03:46.686681 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:46.686683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:46.686910 (+ 227us) server_negotiation.cc:407] Connection header received
0929 19:03:46.687369 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:46.687370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:46.687374 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:46.687433 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:46.688605 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:46.688606 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:46.688607 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:03:46.688610 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:46.688622 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:03:46.688625 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:46.688673 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:46.688718 (+ 45us) server_negotiation.cc:299] Negotiation successful
0929 19:03:46.688735 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 19:03:46.688977 18865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:46.686610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:44329 (local address 127.8.158.66:52933)
0929 19:03:46.686796 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:46.686837 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:46.687284 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:46.687482 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:46.687485 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:46.687493 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:46.688453 (+ 960us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:03:46.688453 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:46.688480 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:03:46.688507 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:46.688672 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:46.688674 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:46.688676 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 19:03:46.688725 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 19:03:46.688775 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
I20260929 19:03:46.689442 18863 heartbeater.cc:347] Connected to a master server at 127.8.158.126:44329
I20260929 19:03:46.689540 18863 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:46.689680 18863 heartbeater.cc:511] Master 127.8.158.126:44329 requested a full tablet report, sending...
I20260929 19:03:46.690016 18544 ts_manager.cc:194] Registered new tserver with Master: 0519336f5e2d459ab344adc10f6c1f60 (127.8.158.66:44757)
I20260929 19:03:46.695057 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 18735
I20260929 19:03:46.695133 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance
I20260929 19:03:46.695266 8825 external_mini_cluster.cc:1428] Setting key 93bee09c2682b9bc901add09d739cf8d
I20260929 19:03:46.696053 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.67:0
--local_ip_for_outbound_sockets=127.8.158.67
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:44329
--builtin_ntp_servers=127.8.158.84:38625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 19:03:46.774936 18866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:46.775149 18866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:46.775171 18866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:46.776249 18866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:46.776281 18866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:46.776306 18866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.67
I20260929 19:03:46.777527 18866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:38625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:44329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.67
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:46.777807 18866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:46.778151 18866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:46.782929 18874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:46.782967 18872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:46.782955 18871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:46.783109 18866 server_base.cc:1034] running on GCE node
I20260929 19:03:46.783550 18866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:46.783739 18866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:46.784883 18866 hybrid_clock.cc:648] HybridClock initialized: now 1790708626784875 us; error 31 us; skew 500 ppm
I20260929 19:03:46.786064 18866 webserver.cc:460] Webserver started at http://127.8.158.67:34147/ using document root <none> and password file <none>
I20260929 19:03:46.786294 18866 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:46.786391 18866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:46.786573 18866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:46.787565 18866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/data/instance:
uuid: "96d08b9989904228bb20ce9f2dc6129b"
format_stamp: "Formatted at 2026-09-29 19:03:46 on dist-test-slave-nmsz"
server_key: "6eb802979c9d51172de5239ddec66a4a"
server_key_iv: "e0e3e16cd595f46c8dd24ecbb0aef428"
server_key_version: "encryptionkey@0"
I20260929 19:03:46.787917 18866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance:
uuid: "96d08b9989904228bb20ce9f2dc6129b"
format_stamp: "Formatted at 2026-09-29 19:03:46 on dist-test-slave-nmsz"
server_key: "6eb802979c9d51172de5239ddec66a4a"
server_key_iv: "e0e3e16cd595f46c8dd24ecbb0aef428"
server_key_version: "encryptionkey@0"
I20260929 19:03:46.789181 18866 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:03:46.789919 18880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:46.790093 18866 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:03:46.790189 18866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/wal
uuid: "96d08b9989904228bb20ce9f2dc6129b"
format_stamp: "Formatted at 2026-09-29 19:03:46 on dist-test-slave-nmsz"
server_key: "6eb802979c9d51172de5239ddec66a4a"
server_key_iv: "e0e3e16cd595f46c8dd24ecbb0aef428"
server_key_version: "encryptionkey@0"
I20260929 19:03:46.790274 18866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:46.793445 18866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:46.793694 18866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:46.793824 18866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:46.794034 18866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:46.794353 18866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:46.794396 18866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:46.794435 18866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:46.794462 18866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:46.799995 18866 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.67:44981
I20260929 19:03:46.800028 18993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.67:44981 every 8 connection(s)
I20260929 19:03:46.800318 18866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
I20260929 19:03:46.803146 18583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:46.800947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:51515 (local address 127.8.158.126:44329)
0929 19:03:46.800989 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:46.800992 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:46.801185 (+ 193us) server_negotiation.cc:407] Connection header received
0929 19:03:46.801630 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:46.801631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:46.801637 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:46.801683 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:46.802900 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:46.802902 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:46.802902 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:03:46.802907 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:46.802922 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:03:46.802926 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:46.802978 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:46.803065 (+ 87us) server_negotiation.cc:299] Negotiation successful
0929 19:03:46.803079 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 19:03:46.803334 18996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:46.800918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:44329 (local address 127.8.158.67:51515)
0929 19:03:46.801071 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:46.801098 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:46.801528 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:46.801784 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:46.801787 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:46.801794 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:46.802755 (+ 961us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:03:46.802755 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:46.802780 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:03:46.802805 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:46.802976 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:46.802980 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:46.802983 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:03:46.803075 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 19:03:46.803130 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":82,"threads_started":1}
I20260929 19:03:46.803853 18994 heartbeater.cc:347] Connected to a master server at 127.8.158.126:44329
I20260929 19:03:46.803927 18994 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:46.804101 18994 heartbeater.cc:511] Master 127.8.158.126:44329 requested a full tablet report, sending...
I20260929 19:03:46.804482 18544 ts_manager.cc:194] Registered new tserver with Master: 96d08b9989904228bb20ce9f2dc6129b (127.8.158.67:44981)
I20260929 19:03:46.810515 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 18866
I20260929 19:03:46.810598 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance
I20260929 19:03:46.810798 8825 external_mini_cluster.cc:1428] Setting key 449228bdb6b77b3d07cf09b7f4ec4060
I20260929 19:03:46.811750 8825 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 19:03:46.813632 18583 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:46.813099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44648 (local address 127.8.158.126:44329)
0929 19:03:46.813178 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:46.813180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:46.813314 (+ 134us) server_negotiation.cc:407] Connection header received
0929 19:03:46.813356 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:46.813356 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:46.813360 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:46.813399 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:46.813525 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44648: BlockingRecv error: recv got EOF from 127.0.0.1:44648 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 19:03:46.813684 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 18604
I20260929 19:03:46.818569 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 18735
I20260929 19:03:46.822962 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 18866
I20260929 19:03:46.826627 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 18513
2026-09-29T19:03:46Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (517 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:03:46 dist-test-slave-nmsz krb5kdc[19004](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:03:46 dist-test-slave-nmsz krb5kdc[19004](info): set up 2 sockets
Sep 29 19:03:46 dist-test-slave-nmsz krb5kdc[19004](info): commencing operation
krb5kdc: starting...
W20260929 19:03:47.863639 8825 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Sep 29 19:03:47 dist-test-slave-nmsz krb5kdc[19004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:03:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:03:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.158.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
I20260929 19:03:47.985255 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:36917
--webserver_interface=127.8.158.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:39903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:36917
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:48.070595 19017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:48.070829 19017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:48.070866 19017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:48.071988 19017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:03:48.072058 19017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:48.072084 19017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:48.072117 19017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:03:48.072141 19017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:03:48.073573 19017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:39903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:36917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:36917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.158.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:48.073837 19017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:48.074070 19017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:48.078896 19022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:48.078859 19025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:48.078878 19023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:48.079026 19017 server_base.cc:1034] running on GCE node
I20260929 19:03:48.079403 19017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:48.079777 19017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:48.080953 19017 hybrid_clock.cc:648] HybridClock initialized: now 1790708628080948 us; error 45 us; skew 500 ppm
Sep 29 19:03:48 dist-test-slave-nmsz krb5kdc[19004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708628, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:48.084018 19017 init.cc:377] Logged in from keytab as kudu/127.8.158.126@KRBTEST.COM (short username kudu)
I20260929 19:03:48.084583 19017 webserver.cc:460] Webserver started at http://127.8.158.126:41131/ using document root <none> and password file <none>
I20260929 19:03:48.084842 19017 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:48.084892 19017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:48.085049 19017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:48.086097 19017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "71b593aaeb47461e9948ef76966eeb68"
format_stamp: "Formatted at 2026-09-29 19:03:48 on dist-test-slave-nmsz"
server_key: "a9e9ebe29b924f2db61b154eaa88f95e"
server_key_iv: "ea8b0c1a2a81fb44b341399c11594e7e"
server_key_version: "encryptionkey@0"
I20260929 19:03:48.086481 19017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "71b593aaeb47461e9948ef76966eeb68"
format_stamp: "Formatted at 2026-09-29 19:03:48 on dist-test-slave-nmsz"
server_key: "a9e9ebe29b924f2db61b154eaa88f95e"
server_key_iv: "ea8b0c1a2a81fb44b341399c11594e7e"
server_key_version: "encryptionkey@0"
I20260929 19:03:48.087886 19017 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:48.088721 19032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:48.088999 19017 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:48.089092 19017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "71b593aaeb47461e9948ef76966eeb68"
format_stamp: "Formatted at 2026-09-29 19:03:48 on dist-test-slave-nmsz"
server_key: "a9e9ebe29b924f2db61b154eaa88f95e"
server_key_iv: "ea8b0c1a2a81fb44b341399c11594e7e"
server_key_version: "encryptionkey@0"
I20260929 19:03:48.089187 19017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:48.104439 19017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:48.106813 19017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:48.106957 19017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:48.110870 19017 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.126:36917
I20260929 19:03:48.110911 19084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.126:36917 every 8 connection(s)
I20260929 19:03:48.111294 19017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:03:48.112396 19085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:48.114745 19085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71b593aaeb47461e9948ef76966eeb68: Bootstrap starting.
I20260929 19:03:48.115314 19085 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71b593aaeb47461e9948ef76966eeb68: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:48.115603 19085 log.cc:824] T 00000000000000000000000000000000 P 71b593aaeb47461e9948ef76966eeb68: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:48.116328 19085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71b593aaeb47461e9948ef76966eeb68: No bootstrap required, opened a new log
I20260929 19:03:48.117920 19085 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71b593aaeb47461e9948ef76966eeb68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71b593aaeb47461e9948ef76966eeb68" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 36917 } }
I20260929 19:03:48.118057 19085 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71b593aaeb47461e9948ef76966eeb68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:48.118079 19085 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71b593aaeb47461e9948ef76966eeb68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71b593aaeb47461e9948ef76966eeb68, State: Initialized, Role: FOLLOWER
I20260929 19:03:48.118257 19085 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71b593aaeb47461e9948ef76966eeb68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71b593aaeb47461e9948ef76966eeb68" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 36917 } }
I20260929 19:03:48.118379 19085 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71b593aaeb47461e9948ef76966eeb68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:03:48.118432 19085 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71b593aaeb47461e9948ef76966eeb68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:03:48.118480 19085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71b593aaeb47461e9948ef76966eeb68 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:48.119311 19085 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71b593aaeb47461e9948ef76966eeb68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71b593aaeb47461e9948ef76966eeb68" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 36917 } }
I20260929 19:03:48.119421 19085 leader_election.cc:302] T 00000000000000000000000000000000 P 71b593aaeb47461e9948ef76966eeb68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71b593aaeb47461e9948ef76966eeb68; no voters:
I20260929 19:03:48.119585 19085 leader_election.cc:288] T 00000000000000000000000000000000 P 71b593aaeb47461e9948ef76966eeb68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:03:48.119614 19088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71b593aaeb47461e9948ef76966eeb68 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:03:48.119802 19088 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71b593aaeb47461e9948ef76966eeb68 [term 1 LEADER]: Becoming Leader. State: Replica: 71b593aaeb47461e9948ef76966eeb68, State: Running, Role: LEADER
I20260929 19:03:48.119846 19085 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71b593aaeb47461e9948ef76966eeb68 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:03:48.119886 19088 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71b593aaeb47461e9948ef76966eeb68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71b593aaeb47461e9948ef76966eeb68" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 36917 } }
I20260929 19:03:48.120316 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 19017
I20260929 19:03:48.120424 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:03:48.120630 8825 external_mini_cluster.cc:1428] Setting key 83c3c1c8b1b865079c313f6480a2d374
I20260929 19:03:48.120985 19090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71b593aaeb47461e9948ef76966eeb68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71b593aaeb47461e9948ef76966eeb68. Latest consensus state: current_term: 1 leader_uuid: "71b593aaeb47461e9948ef76966eeb68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71b593aaeb47461e9948ef76966eeb68" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 36917 } } }
I20260929 19:03:48.121081 19090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71b593aaeb47461e9948ef76966eeb68 [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:48.121413 19089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71b593aaeb47461e9948ef76966eeb68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71b593aaeb47461e9948ef76966eeb68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71b593aaeb47461e9948ef76966eeb68" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 36917 } } }
I20260929 19:03:48.121624 19089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71b593aaeb47461e9948ef76966eeb68 [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:48.121773 19104 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:03:48.122170 19104 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:03:48.126117 19104 catalog_manager.cc:1347] Generated new cluster ID: 8b28740b8dd84db69d7f0f1549e79e63
I20260929 19:03:48.126168 19104 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 19:03:48 dist-test-slave-nmsz krb5kdc[19004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:03:48.132946 19105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:48.121130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38154 (local address 127.8.158.126:36917)
0929 19:03:48.121520 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:48.121530 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:48.121568 (+ 38us) server_negotiation.cc:407] Connection header received
0929 19:03:48.121705 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:48.121714 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:48.121772 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:48.121940 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:48.122715 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:48.123731 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:48.124770 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:48.125183 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:48.125277 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:48.128078 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:48.128082 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:48.128083 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:48.128098 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:48.130789 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:48.131220 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:48.131223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:48.131224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:48.131353 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:48.131563 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:48.131564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:48.131564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:48.131892 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:48.132050 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:48.132540 (+ 490us) server_negotiation.cc:299] Negotiation successful
0929 19:03:48.132701 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":193,"threads_started":1}
I20260929 19:03:48.148165 19104 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:03:48.148787 19104 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:03:48.164516 19104 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71b593aaeb47461e9948ef76966eeb68: Generated new TSK 0
I20260929 19:03:48.164714 19104 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.158.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab.
Entry for principal kudu/127.8.158.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab.
I20260929 19:03:48.193418 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.65:0
--local_ip_for_outbound_sockets=127.8.158.65
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:36917
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab
--principal=kudu/127.8.158.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:39903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:48.271800 19111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:48.271984 19111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:48.272006 19111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:48.273097 19111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:48.273136 19111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:48.273161 19111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.65
I20260929 19:03:48.274585 19111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:39903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab
--principal=kudu/127.8.158.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:36917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.65
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:48.274793 19111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:48.275012 19111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:48.279865 19116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:48.279999 19111 server_base.cc:1034] running on GCE node
W20260929 19:03:48.279873 19119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:48.279901 19117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:48.280332 19111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:48.280591 19111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:48.281729 19111 hybrid_clock.cc:648] HybridClock initialized: now 1790708628281715 us; error 30 us; skew 500 ppm
Sep 29 19:03:48 dist-test-slave-nmsz krb5kdc[19004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708628, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:48.284219 19111 init.cc:377] Logged in from keytab as kudu/127.8.158.65@KRBTEST.COM (short username kudu)
I20260929 19:03:48.284574 19111 webserver.cc:460] Webserver started at http://127.8.158.65:36857/ using document root <none> and password file <none>
I20260929 19:03:48.284802 19111 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:48.284871 19111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:48.284999 19111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:48.286006 19111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/data/instance:
uuid: "61e6d436328547f18c8716ad7fbe1d11"
format_stamp: "Formatted at 2026-09-29 19:03:48 on dist-test-slave-nmsz"
server_key: "77f5d92a88d74c851f7c545354761d43"
server_key_iv: "09454c55990aef6ae2a9197e8d9da1cf"
server_key_version: "encryptionkey@0"
I20260929 19:03:48.286424 19111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance:
uuid: "61e6d436328547f18c8716ad7fbe1d11"
format_stamp: "Formatted at 2026-09-29 19:03:48 on dist-test-slave-nmsz"
server_key: "77f5d92a88d74c851f7c545354761d43"
server_key_iv: "09454c55990aef6ae2a9197e8d9da1cf"
server_key_version: "encryptionkey@0"
I20260929 19:03:48.287825 19111 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:03:48.288596 19126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:48.288790 19111 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:03:48.288887 19111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/wal
uuid: "61e6d436328547f18c8716ad7fbe1d11"
format_stamp: "Formatted at 2026-09-29 19:03:48 on dist-test-slave-nmsz"
server_key: "77f5d92a88d74c851f7c545354761d43"
server_key_iv: "09454c55990aef6ae2a9197e8d9da1cf"
server_key_version: "encryptionkey@0"
I20260929 19:03:48.288975 19111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:48.296870 19111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:48.298763 19111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:48.298918 19111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:48.299134 19111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:48.299435 19111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:48.299497 19111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:48.299535 19111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:48.299563 19111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:48.304963 19111 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.65:32937
I20260929 19:03:48.305027 19239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.65:32937 every 8 connection(s)
I20260929 19:03:48.305330 19111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
I20260929 19:03:48.307761 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 19111
I20260929 19:03:48.307837 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance
I20260929 19:03:48.308044 8825 external_mini_cluster.cc:1428] Setting key 5ddff300a2fd66af35567e797e5c3769
Sep 29 19:03:48 dist-test-slave-nmsz krb5kdc[19004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708628, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:03:48.315661 19105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:48.306018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:40659 (local address 127.8.158.126:36917)
0929 19:03:48.306076 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:48.306078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:48.306276 (+ 198us) server_negotiation.cc:407] Connection header received
0929 19:03:48.306799 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:48.306800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:48.306813 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:48.306867 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:48.307756 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:48.308497 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:48.310003 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:48.310389 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:48.310465 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:48.312691 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:48.312692 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:48.312693 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:48.312702 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:48.314355 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:48.314733 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:48.314735 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:48.314735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:48.314798 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:48.315023 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:48.315024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:48.315024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:48.315177 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:48.315254 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:48.315475 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 19:03:48.315560 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 19:03:48.315696 19242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:48.305999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:36917 (local address 127.8.158.65:40659)
0929 19:03:48.306167 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:48.306199 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:48.306707 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:48.306931 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:48.306934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:48.307474 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:48.307643 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:48.307647 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:48.308641 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:48.308643 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:48.309914 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:48.309917 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:48.310477 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:48.310478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:48.310582 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:48.311090 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:48.311099 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:48.312576 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:48.314430 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:48.314435 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:48.314437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:48.314642 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:48.314866 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:48.314867 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:48.314867 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:48.314957 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:48.315247 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:48.315250 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:48.315348 (+ 98us) client_negotiation.cc:769] Sending connection context
0929 19:03:48.315427 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 19:03:48.315535 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":85,"threads_started":1}
I20260929 19:03:48.316202 19240 heartbeater.cc:347] Connected to a master server at 127.8.158.126:36917
I20260929 19:03:48.316313 19240 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:48.316476 19240 heartbeater.cc:511] Master 127.8.158.126:36917 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.158.66@KRBTEST.COM; defaulting to no policy
I20260929 19:03:48.316989 19049 ts_manager.cc:194] Registered new tserver with Master: 61e6d436328547f18c8716ad7fbe1d11 (127.8.158.65:32937)
I20260929 19:03:48.317524 19049 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.65@KRBTEST.COM'} at 127.8.158.65:40659
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab.
Entry for principal kudu/127.8.158.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab.
I20260929 19:03:48.334116 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.66:0
--local_ip_for_outbound_sockets=127.8.158.66
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:36917
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab
--principal=kudu/127.8.158.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:39903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:48.414130 19245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:48.414381 19245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:48.414423 19245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:48.415584 19245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:48.415652 19245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:48.415694 19245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.66
I20260929 19:03:48.417133 19245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:39903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab
--principal=kudu/127.8.158.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:36917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.66
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:48.417392 19245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:48.417610 19245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:48.422586 19253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:48.422622 19251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:48.422602 19250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:48.422748 19245 server_base.cc:1034] running on GCE node
I20260929 19:03:48.422996 19245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:48.423210 19245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:48.424371 19245 hybrid_clock.cc:648] HybridClock initialized: now 1790708628424365 us; error 35 us; skew 500 ppm
Sep 29 19:03:48 dist-test-slave-nmsz krb5kdc[19004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708628, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:48.426819 19245 init.cc:377] Logged in from keytab as kudu/127.8.158.66@KRBTEST.COM (short username kudu)
I20260929 19:03:48.427271 19245 webserver.cc:460] Webserver started at http://127.8.158.66:44841/ using document root <none> and password file <none>
I20260929 19:03:48.427488 19245 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:48.427554 19245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:48.427711 19245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:48.428715 19245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/data/instance:
uuid: "11730347cf9f4df3a1fe83c40cef0081"
format_stamp: "Formatted at 2026-09-29 19:03:48 on dist-test-slave-nmsz"
server_key: "cccd070b7c2c8c74ed9c95383252ce89"
server_key_iv: "42969da4c0c5be03ac2afd7ff2f23c8c"
server_key_version: "encryptionkey@0"
I20260929 19:03:48.429062 19245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance:
uuid: "11730347cf9f4df3a1fe83c40cef0081"
format_stamp: "Formatted at 2026-09-29 19:03:48 on dist-test-slave-nmsz"
server_key: "cccd070b7c2c8c74ed9c95383252ce89"
server_key_iv: "42969da4c0c5be03ac2afd7ff2f23c8c"
server_key_version: "encryptionkey@0"
I20260929 19:03:48.430475 19245 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 19:03:48.431382 19260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:48.431674 19245 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:48.431792 19245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/wal
uuid: "11730347cf9f4df3a1fe83c40cef0081"
format_stamp: "Formatted at 2026-09-29 19:03:48 on dist-test-slave-nmsz"
server_key: "cccd070b7c2c8c74ed9c95383252ce89"
server_key_iv: "42969da4c0c5be03ac2afd7ff2f23c8c"
server_key_version: "encryptionkey@0"
I20260929 19:03:48.431886 19245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:48.471621 19245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:48.473934 19245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:48.474215 19245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:48.474510 19245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:48.474828 19245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:48.474884 19245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:48.474925 19245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:48.474956 19245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:48.480777 19245 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.66:37191
I20260929 19:03:48.480816 19373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.66:37191 every 8 connection(s)
I20260929 19:03:48.481155 19245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
Sep 29 19:03:48 dist-test-slave-nmsz krb5kdc[19004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708628, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.66@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:03:48.489151 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 19245
I20260929 19:03:48.489275 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance
I20260929 19:03:48.489468 8825 external_mini_cluster.cc:1428] Setting key e6e72d215606a65ec7b6bf121878e4a3
I20260929 19:03:48.490268 19105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:48.481874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:54675 (local address 127.8.158.126:36917)
0929 19:03:48.481944 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:48.481947 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:48.482138 (+ 191us) server_negotiation.cc:407] Connection header received
0929 19:03:48.482721 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:48.482723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:48.482735 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:48.482791 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:48.483721 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:48.484221 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:48.485224 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:48.485420 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:48.485507 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:48.487591 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:48.487594 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:48.487594 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:48.487602 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:48.488799 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:48.489238 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:48.489240 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:48.489240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:48.489300 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:48.489553 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:48.489554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:48.489554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:48.489670 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:48.489749 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:48.490038 (+ 289us) server_negotiation.cc:299] Negotiation successful
0929 19:03:48.490149 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 19:03:48.490365 19376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:48.481848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:36917 (local address 127.8.158.66:54675)
0929 19:03:48.482003 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:48.482049 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:48.482575 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:48.482861 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:48.482864 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:48.483425 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:48.483615 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:48.483620 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:48.484317 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:48.484318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:48.485130 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:48.485132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:48.485515 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:48.485517 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:48.485641 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:48.486146 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:48.486153 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:48.487494 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:48.488895 (+ 1401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:48.488901 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:48.488903 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:48.489145 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:48.489380 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:48.489380 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:48.489381 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:48.489483 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:48.489740 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:48.489746 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:48.489898 (+ 152us) client_negotiation.cc:769] Sending connection context
0929 19:03:48.490008 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 19:03:48.490163 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":79,"threads_started":1}
I20260929 19:03:48.491043 19374 heartbeater.cc:347] Connected to a master server at 127.8.158.126:36917
I20260929 19:03:48.491161 19374 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:48.491379 19374 heartbeater.cc:511] Master 127.8.158.126:36917 requested a full tablet report, sending...
I20260929 19:03:48.491776 19049 ts_manager.cc:194] Registered new tserver with Master: 11730347cf9f4df3a1fe83c40cef0081 (127.8.158.66:37191)
I20260929 19:03:48.492116 19049 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.66@KRBTEST.COM'} at 127.8.158.66:54675
WARNING: no policy specified for kudu/127.8.158.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab.
Entry for principal kudu/127.8.158.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab.
I20260929 19:03:48.514784 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.67:0
--local_ip_for_outbound_sockets=127.8.158.67
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:36917
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab
--principal=kudu/127.8.158.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:39903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:48.592608 19379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:48.592834 19379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:48.592857 19379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:48.593954 19379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:48.593995 19379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:48.594050 19379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.67
I20260929 19:03:48.595466 19379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:39903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab
--principal=kudu/127.8.158.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:36917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.67
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:03:48.595687 19379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:48.595913 19379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:48.600639 19387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:48.600772 19379 server_base.cc:1034] running on GCE node
W20260929 19:03:48.600821 19385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:48.600658 19384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:48.601087 19379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:48.601336 19379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:48.602489 19379 hybrid_clock.cc:648] HybridClock initialized: now 1790708628602476 us; error 26 us; skew 500 ppm
Sep 29 19:03:48 dist-test-slave-nmsz krb5kdc[19004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708628, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:48.604794 19379 init.cc:377] Logged in from keytab as kudu/127.8.158.67@KRBTEST.COM (short username kudu)
I20260929 19:03:48.605247 19379 webserver.cc:460] Webserver started at http://127.8.158.67:43267/ using document root <none> and password file <none>
I20260929 19:03:48.605482 19379 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:48.605549 19379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:48.605698 19379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:48.606822 19379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/data/instance:
uuid: "fb41bf1b34464010a1aba0b8438453a2"
format_stamp: "Formatted at 2026-09-29 19:03:48 on dist-test-slave-nmsz"
server_key: "6d0d357dbb6c7ce94788fc7b696123cb"
server_key_iv: "6f1c3aa3acd880dc34c141997e30201b"
server_key_version: "encryptionkey@0"
I20260929 19:03:48.607182 19379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance:
uuid: "fb41bf1b34464010a1aba0b8438453a2"
format_stamp: "Formatted at 2026-09-29 19:03:48 on dist-test-slave-nmsz"
server_key: "6d0d357dbb6c7ce94788fc7b696123cb"
server_key_iv: "6f1c3aa3acd880dc34c141997e30201b"
server_key_version: "encryptionkey@0"
I20260929 19:03:48.608623 19379 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:03:48.609381 19394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:48.609560 19379 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:48.609655 19379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/wal
uuid: "fb41bf1b34464010a1aba0b8438453a2"
format_stamp: "Formatted at 2026-09-29 19:03:48 on dist-test-slave-nmsz"
server_key: "6d0d357dbb6c7ce94788fc7b696123cb"
server_key_iv: "6f1c3aa3acd880dc34c141997e30201b"
server_key_version: "encryptionkey@0"
I20260929 19:03:48.609741 19379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:48.618396 19379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:48.621160 19379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:48.621356 19379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:48.621610 19379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:48.621930 19379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:48.621966 19379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:48.621989 19379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:48.622041 19379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:48.627997 19379 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.67:45345
I20260929 19:03:48.628083 19507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.67:45345 every 8 connection(s)
I20260929 19:03:48.628374 19379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
I20260929 19:03:48.628940 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 19379
I20260929 19:03:48.629058 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance
I20260929 19:03:48.629269 8825 external_mini_cluster.cc:1428] Setting key 47271f57914656c36da2d651434b09e1
Sep 29 19:03:48 dist-test-slave-nmsz krb5kdc[19004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708628, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.67@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:03:48.637459 19105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:48.629109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:51383 (local address 127.8.158.126:36917)
0929 19:03:48.629162 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:48.629165 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:48.629467 (+ 302us) server_negotiation.cc:407] Connection header received
0929 19:03:48.630145 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:48.630146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:48.630156 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:48.630206 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:48.631135 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:48.631628 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:48.632550 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:48.632730 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:48.632789 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:48.634983 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:48.634984 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:48.634985 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:48.634991 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:48.636327 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:48.636713 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:48.636714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:48.636714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:48.636773 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:48.636985 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:48.636986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:48.636987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:48.637091 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:48.637142 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:48.637344 (+ 202us) server_negotiation.cc:299] Negotiation successful
0929 19:03:48.637400 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 19:03:48.637657 19510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:48.629174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:36917 (local address 127.8.158.67:51383)
0929 19:03:48.629329 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:48.629368 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:48.630029 (+ 661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:48.630256 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:48.630260 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:48.630858 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:48.631036 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:48.631040 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:48.631715 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:48.631716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:48.632472 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:48.632473 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:48.632804 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:48.632805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:48.632897 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:48.633421 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:48.633429 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:48.634899 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:48.636415 (+ 1516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:48.636420 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:48.636422 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:48.636628 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:48.636841 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:48.636841 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:48.636841 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:48.636930 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:48.637145 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:48.637149 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:48.637254 (+ 105us) client_negotiation.cc:769] Sending connection context
0929 19:03:48.637331 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 19:03:48.637447 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":91,"threads_started":1}
I20260929 19:03:48.638167 19508 heartbeater.cc:347] Connected to a master server at 127.8.158.126:36917
I20260929 19:03:48.638293 19508 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:48.638516 19508 heartbeater.cc:511] Master 127.8.158.126:36917 requested a full tablet report, sending...
I20260929 19:03:48.638923 19049 ts_manager.cc:194] Registered new tserver with Master: fb41bf1b34464010a1aba0b8438453a2 (127.8.158.67:45345)
I20260929 19:03:48.639308 19049 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.67@KRBTEST.COM'} at 127.8.158.67:51383
I20260929 19:03:48.641412 8825 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:03:48.648651 19105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:48.642724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38166 (local address 127.8.158.126:36917)
0929 19:03:48.642789 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:48.642791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:48.642887 (+ 96us) server_negotiation.cc:407] Connection header received
0929 19:03:48.642997 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:48.642998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:48.643009 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:48.643058 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:48.643598 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:48.644064 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:48.644836 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:48.645062 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:48.645137 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:48.646257 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:48.646259 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:48.646259 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:48.646266 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:48.647495 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:48.647830 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:48.647831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:48.647831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:48.647883 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:48.648095 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:48.648096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:48.648096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:48.648241 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:48.648342 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:48.648450 (+ 108us) server_negotiation.cc:299] Negotiation successful
0929 19:03:48.648540 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 19:03:48.650243 19049 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38166:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:03:48.651901 19049 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:03:48.657994 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:48.653926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.66:37191 (local address 127.0.0.1:46574)
0929 19:03:48.654062 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:48.654099 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:48.654158 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:48.654495 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:48.654498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:48.654501 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:48.654569 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:48.654571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:48.655884 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:48.655885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:48.656735 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:48.656738 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:48.657692 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:48.657692 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:48.657779 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:48.657795 (+ 16us) client_negotiation.cc:769] Sending connection context
0929 19:03:48.657863 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 19:03:48.657914 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":24,"thread_start_us":87,"threads_started":1}
I20260929 19:03:48.658020 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:48.654012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46574 (local address 127.8.158.66:37191)
0929 19:03:48.654190 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:48.654195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:48.654215 (+ 20us) server_negotiation.cc:407] Connection header received
0929 19:03:48.654258 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:48.654262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:48.654279 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:48.654387 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:48.654658 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:48.655773 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:48.656865 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:48.657594 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:48.657672 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:48.657738 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:48.657891 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 19:03:48.657944 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":113,"threads_started":1}
I20260929 19:03:48.658181 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:48.654283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.67:45345 (local address 127.0.0.1:47310)
0929 19:03:48.654605 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:48.654614 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:48.654683 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:48.654849 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:48.654850 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:48.654852 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:48.654904 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:48.654905 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:48.655831 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:48.655832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:48.657181 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:48.657183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:48.657940 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:48.657941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:48.657996 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:48.658000 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:03:48.658049 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 19:03:48.658092 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":277,"threads_started":1}
I20260929 19:03:48.658187 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:48.654148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:32937 (local address 127.0.0.1:43964)
0929 19:03:48.654236 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:48.654247 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:48.654303 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:48.654746 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:48.654747 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:48.654749 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:48.654780 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:48.654782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:48.656052 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:48.656052 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:48.657094 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:48.657097 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:48.657980 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:48.657981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:48.658047 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:48.658052 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 19:03:48.658084 (+ 32us) client_negotiation.cc:240] Negotiation successful
0929 19:03:48.658119 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75,"thread_start_us":51,"threads_started":1}
I20260929 19:03:48.658447 19521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:48.654430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47310 (local address 127.8.158.67:45345)
0929 19:03:48.654589 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:48.654593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:48.654688 (+ 95us) server_negotiation.cc:407] Connection header received
0929 19:03:48.654749 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:48.654751 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:48.654765 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:48.654839 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:48.654986 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:48.655722 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:48.657241 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:48.657782 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:48.657852 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:48.657903 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:48.658283 (+ 380us) server_negotiation.cc:299] Negotiation successful
0929 19:03:48.658353 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":74,"threads_started":1}
I20260929 19:03:48.658548 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:48.654379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43964 (local address 127.8.158.65:32937)
0929 19:03:48.654562 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:48.654566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:48.654575 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:03:48.654615 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:48.654617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:48.654634 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:48.654848 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:48.654979 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:48.655951 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:48.657162 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:48.657878 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:48.658346 (+ 468us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:48.658413 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:48.658437 (+ 24us) server_negotiation.cc:299] Negotiation successful
0929 19:03:48.658480 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":88,"threads_started":1}
I20260929 19:03:48.658885 19308 tablet_service.cc:1467] Processing CreateTablet for tablet 0f8e652b24374d26a6edb03695c61feb (DEFAULT_TABLE table=test-table [id=76d98b25b0a14378834cdc2325c5a97e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:48.659046 19442 tablet_service.cc:1467] Processing CreateTablet for tablet 0f8e652b24374d26a6edb03695c61feb (DEFAULT_TABLE table=test-table [id=76d98b25b0a14378834cdc2325c5a97e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:48.659246 19308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f8e652b24374d26a6edb03695c61feb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:48.659173 19174 tablet_service.cc:1467] Processing CreateTablet for tablet 0f8e652b24374d26a6edb03695c61feb (DEFAULT_TABLE table=test-table [id=76d98b25b0a14378834cdc2325c5a97e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:48.659356 19442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f8e652b24374d26a6edb03695c61feb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:48.659438 19174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f8e652b24374d26a6edb03695c61feb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:48.661610 19522 tablet_bootstrap.cc:492] T 0f8e652b24374d26a6edb03695c61feb P 11730347cf9f4df3a1fe83c40cef0081: Bootstrap starting.
I20260929 19:03:48.662441 19523 tablet_bootstrap.cc:492] T 0f8e652b24374d26a6edb03695c61feb P fb41bf1b34464010a1aba0b8438453a2: Bootstrap starting.
I20260929 19:03:48.662624 19522 tablet_bootstrap.cc:654] T 0f8e652b24374d26a6edb03695c61feb P 11730347cf9f4df3a1fe83c40cef0081: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:48.663097 19523 tablet_bootstrap.cc:654] T 0f8e652b24374d26a6edb03695c61feb P fb41bf1b34464010a1aba0b8438453a2: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:48.663187 19522 log.cc:824] T 0f8e652b24374d26a6edb03695c61feb P 11730347cf9f4df3a1fe83c40cef0081: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:48.663694 19523 log.cc:824] T 0f8e652b24374d26a6edb03695c61feb P fb41bf1b34464010a1aba0b8438453a2: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:48.663969 19522 tablet_bootstrap.cc:492] T 0f8e652b24374d26a6edb03695c61feb P 11730347cf9f4df3a1fe83c40cef0081: No bootstrap required, opened a new log
I20260929 19:03:48.664017 19524 tablet_bootstrap.cc:492] T 0f8e652b24374d26a6edb03695c61feb P 61e6d436328547f18c8716ad7fbe1d11: Bootstrap starting.
I20260929 19:03:48.664042 19522 ts_tablet_manager.cc:1397] T 0f8e652b24374d26a6edb03695c61feb P 11730347cf9f4df3a1fe83c40cef0081: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:03:48.664376 19523 tablet_bootstrap.cc:492] T 0f8e652b24374d26a6edb03695c61feb P fb41bf1b34464010a1aba0b8438453a2: No bootstrap required, opened a new log
I20260929 19:03:48.664458 19523 ts_tablet_manager.cc:1397] T 0f8e652b24374d26a6edb03695c61feb P fb41bf1b34464010a1aba0b8438453a2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260929 19:03:48.664650 19524 tablet_bootstrap.cc:654] T 0f8e652b24374d26a6edb03695c61feb P 61e6d436328547f18c8716ad7fbe1d11: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:48.665278 19524 log.cc:824] T 0f8e652b24374d26a6edb03695c61feb P 61e6d436328547f18c8716ad7fbe1d11: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:48.666078 19524 tablet_bootstrap.cc:492] T 0f8e652b24374d26a6edb03695c61feb P 61e6d436328547f18c8716ad7fbe1d11: No bootstrap required, opened a new log
I20260929 19:03:48.666148 19524 ts_tablet_manager.cc:1397] T 0f8e652b24374d26a6edb03695c61feb P 61e6d436328547f18c8716ad7fbe1d11: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:03:48.666090 19522 raft_consensus.cc:348] T 0f8e652b24374d26a6edb03695c61feb P 11730347cf9f4df3a1fe83c40cef0081 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11730347cf9f4df3a1fe83c40cef0081" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 37191 } } peers { permanent_uuid: "61e6d436328547f18c8716ad7fbe1d11" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 32937 } } peers { permanent_uuid: "fb41bf1b34464010a1aba0b8438453a2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45345 } }
I20260929 19:03:48.666270 19522 raft_consensus.cc:374] T 0f8e652b24374d26a6edb03695c61feb P 11730347cf9f4df3a1fe83c40cef0081 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:48.666297 19522 raft_consensus.cc:729] T 0f8e652b24374d26a6edb03695c61feb P 11730347cf9f4df3a1fe83c40cef0081 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11730347cf9f4df3a1fe83c40cef0081, State: Initialized, Role: FOLLOWER
I20260929 19:03:48.666301 19523 raft_consensus.cc:348] T 0f8e652b24374d26a6edb03695c61feb P fb41bf1b34464010a1aba0b8438453a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11730347cf9f4df3a1fe83c40cef0081" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 37191 } } peers { permanent_uuid: "61e6d436328547f18c8716ad7fbe1d11" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 32937 } } peers { permanent_uuid: "fb41bf1b34464010a1aba0b8438453a2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45345 } }
I20260929 19:03:48.666461 19522 consensus_queue.cc:260] T 0f8e652b24374d26a6edb03695c61feb P 11730347cf9f4df3a1fe83c40cef0081 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11730347cf9f4df3a1fe83c40cef0081" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 37191 } } peers { permanent_uuid: "61e6d436328547f18c8716ad7fbe1d11" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 32937 } } peers { permanent_uuid: "fb41bf1b34464010a1aba0b8438453a2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45345 } }
I20260929 19:03:48.666499 19523 raft_consensus.cc:374] T 0f8e652b24374d26a6edb03695c61feb P fb41bf1b34464010a1aba0b8438453a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:48.666543 19523 raft_consensus.cc:729] T 0f8e652b24374d26a6edb03695c61feb P fb41bf1b34464010a1aba0b8438453a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb41bf1b34464010a1aba0b8438453a2, State: Initialized, Role: FOLLOWER
I20260929 19:03:48.666688 19523 consensus_queue.cc:260] T 0f8e652b24374d26a6edb03695c61feb P fb41bf1b34464010a1aba0b8438453a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11730347cf9f4df3a1fe83c40cef0081" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 37191 } } peers { permanent_uuid: "61e6d436328547f18c8716ad7fbe1d11" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 32937 } } peers { permanent_uuid: "fb41bf1b34464010a1aba0b8438453a2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45345 } }
I20260929 19:03:48.666750 19374 heartbeater.cc:503] Master 127.8.158.126:36917 was elected leader, sending a full tablet report...
I20260929 19:03:48.667001 19508 heartbeater.cc:503] Master 127.8.158.126:36917 was elected leader, sending a full tablet report...
I20260929 19:03:48.667003 19523 ts_tablet_manager.cc:1428] T 0f8e652b24374d26a6edb03695c61feb P fb41bf1b34464010a1aba0b8438453a2: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260929 19:03:48.666741 19522 ts_tablet_manager.cc:1428] T 0f8e652b24374d26a6edb03695c61feb P 11730347cf9f4df3a1fe83c40cef0081: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 19:03:48.668159 19524 raft_consensus.cc:348] T 0f8e652b24374d26a6edb03695c61feb P 61e6d436328547f18c8716ad7fbe1d11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11730347cf9f4df3a1fe83c40cef0081" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 37191 } } peers { permanent_uuid: "61e6d436328547f18c8716ad7fbe1d11" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 32937 } } peers { permanent_uuid: "fb41bf1b34464010a1aba0b8438453a2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45345 } }
I20260929 19:03:48.668275 19524 raft_consensus.cc:374] T 0f8e652b24374d26a6edb03695c61feb P 61e6d436328547f18c8716ad7fbe1d11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:48.668295 19524 raft_consensus.cc:729] T 0f8e652b24374d26a6edb03695c61feb P 61e6d436328547f18c8716ad7fbe1d11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61e6d436328547f18c8716ad7fbe1d11, State: Initialized, Role: FOLLOWER
I20260929 19:03:48.668394 19524 consensus_queue.cc:260] T 0f8e652b24374d26a6edb03695c61feb P 61e6d436328547f18c8716ad7fbe1d11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11730347cf9f4df3a1fe83c40cef0081" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 37191 } } peers { permanent_uuid: "61e6d436328547f18c8716ad7fbe1d11" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 32937 } } peers { permanent_uuid: "fb41bf1b34464010a1aba0b8438453a2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45345 } }
I20260929 19:03:48.668607 19524 ts_tablet_manager.cc:1428] T 0f8e652b24374d26a6edb03695c61feb P 61e6d436328547f18c8716ad7fbe1d11: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:03:48.668663 19240 heartbeater.cc:503] Master 127.8.158.126:36917 was elected leader, sending a full tablet report...
W20260929 19:03:48.731787 19375 tablet.cc:2369] T 0f8e652b24374d26a6edb03695c61feb P 11730347cf9f4df3a1fe83c40cef0081: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:03:48.751570 19529 raft_consensus.cc:482] T 0f8e652b24374d26a6edb03695c61feb P fb41bf1b34464010a1aba0b8438453a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:03:48.751698 19529 raft_consensus.cc:504] T 0f8e652b24374d26a6edb03695c61feb P fb41bf1b34464010a1aba0b8438453a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11730347cf9f4df3a1fe83c40cef0081" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 37191 } } peers { permanent_uuid: "61e6d436328547f18c8716ad7fbe1d11" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 32937 } } peers { permanent_uuid: "fb41bf1b34464010a1aba0b8438453a2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45345 } }
I20260929 19:03:48.752134 19529 leader_election.cc:288] T 0f8e652b24374d26a6edb03695c61feb P fb41bf1b34464010a1aba0b8438453a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11730347cf9f4df3a1fe83c40cef0081 (127.8.158.66:37191), 61e6d436328547f18c8716ad7fbe1d11 (127.8.158.65:32937)
I20260929 19:03:48.755110 19510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:48.752289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.66:37191 (local address 127.8.158.67:60639)
0929 19:03:48.752367 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:48.752381 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:48.752426 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:48.752596 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:48.752597 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:48.752599 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:48.752674 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:48.752675 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:48.753531 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:48.753533 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:48.754272 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:48.754274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:48.754846 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:48.754847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:48.754933 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:48.754937 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:03:48.754989 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 19:03:48.755018 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 19:03:48.755143 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:48.752370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:60639 (local address 127.8.158.66:37191)
0929 19:03:48.752430 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:48.752432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:48.752479 (+ 47us) server_negotiation.cc:407] Connection header received
0929 19:03:48.752507 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:48.752508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:48.752518 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:48.752574 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:48.752755 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:48.753440 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:48.754370 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:48.754766 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:48.754822 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:48.754829 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:48.755024 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 19:03:48.755072 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 19:03:48.755195 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:48.752552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:49157 (local address 127.8.158.65:32937)
0929 19:03:48.752596 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:48.752598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:48.752788 (+ 190us) server_negotiation.cc:407] Connection header received
0929 19:03:48.752823 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:48.752824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:48.752832 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:48.752868 (+ 36us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:48.753009 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:48.753443 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:48.754436 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:48.754831 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:48.754901 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:48.754910 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:48.755083 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 19:03:48.755124 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 19:03:48.755223 19531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:48.752448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:32937 (local address 127.8.158.67:49157)
0929 19:03:48.752675 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:48.752687 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:48.752773 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:48.752877 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:48.752877 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:48.752879 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:48.752945 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:48.752947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:48.753544 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:48.753545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:48.754285 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:48.754287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:48.754926 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:48.754926 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:48.754997 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:48.755002 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 19:03:48.755060 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 19:03:48.755094 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":145,"threads_started":1}
I20260929 19:03:48.755635 19194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f8e652b24374d26a6edb03695c61feb" candidate_uuid: "fb41bf1b34464010a1aba0b8438453a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61e6d436328547f18c8716ad7fbe1d11" is_pre_election: true
I20260929 19:03:48.755636 19328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f8e652b24374d26a6edb03695c61feb" candidate_uuid: "fb41bf1b34464010a1aba0b8438453a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11730347cf9f4df3a1fe83c40cef0081" is_pre_election: true
I20260929 19:03:48.755774 19328 raft_consensus.cc:2454] T 0f8e652b24374d26a6edb03695c61feb P 11730347cf9f4df3a1fe83c40cef0081 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb41bf1b34464010a1aba0b8438453a2 in term 0.
I20260929 19:03:48.755792 19194 raft_consensus.cc:2454] T 0f8e652b24374d26a6edb03695c61feb P 61e6d436328547f18c8716ad7fbe1d11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb41bf1b34464010a1aba0b8438453a2 in term 0.
I20260929 19:03:48.755973 19395 leader_election.cc:302] T 0f8e652b24374d26a6edb03695c61feb P fb41bf1b34464010a1aba0b8438453a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11730347cf9f4df3a1fe83c40cef0081, fb41bf1b34464010a1aba0b8438453a2; no voters:
I20260929 19:03:48.756129 19529 raft_consensus.cc:2781] T 0f8e652b24374d26a6edb03695c61feb P fb41bf1b34464010a1aba0b8438453a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:03:48.756196 19529 raft_consensus.cc:482] T 0f8e652b24374d26a6edb03695c61feb P fb41bf1b34464010a1aba0b8438453a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:03:48.756227 19529 raft_consensus.cc:3037] T 0f8e652b24374d26a6edb03695c61feb P fb41bf1b34464010a1aba0b8438453a2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:48.757001 19529 raft_consensus.cc:504] T 0f8e652b24374d26a6edb03695c61feb P fb41bf1b34464010a1aba0b8438453a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11730347cf9f4df3a1fe83c40cef0081" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 37191 } } peers { permanent_uuid: "61e6d436328547f18c8716ad7fbe1d11" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 32937 } } peers { permanent_uuid: "fb41bf1b34464010a1aba0b8438453a2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45345 } }
I20260929 19:03:48.757133 19529 leader_election.cc:288] T 0f8e652b24374d26a6edb03695c61feb P fb41bf1b34464010a1aba0b8438453a2 [CANDIDATE]: Term 1 election: Requested vote from peers 11730347cf9f4df3a1fe83c40cef0081 (127.8.158.66:37191), 61e6d436328547f18c8716ad7fbe1d11 (127.8.158.65:32937)
I20260929 19:03:48.757336 19194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f8e652b24374d26a6edb03695c61feb" candidate_uuid: "fb41bf1b34464010a1aba0b8438453a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61e6d436328547f18c8716ad7fbe1d11"
I20260929 19:03:48.757336 19328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f8e652b24374d26a6edb03695c61feb" candidate_uuid: "fb41bf1b34464010a1aba0b8438453a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11730347cf9f4df3a1fe83c40cef0081"
I20260929 19:03:48.757418 19194 raft_consensus.cc:3037] T 0f8e652b24374d26a6edb03695c61feb P 61e6d436328547f18c8716ad7fbe1d11 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:48.757418 19328 raft_consensus.cc:3037] T 0f8e652b24374d26a6edb03695c61feb P 11730347cf9f4df3a1fe83c40cef0081 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:48.758275 19328 raft_consensus.cc:2454] T 0f8e652b24374d26a6edb03695c61feb P 11730347cf9f4df3a1fe83c40cef0081 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb41bf1b34464010a1aba0b8438453a2 in term 1.
I20260929 19:03:48.758275 19194 raft_consensus.cc:2454] T 0f8e652b24374d26a6edb03695c61feb P 61e6d436328547f18c8716ad7fbe1d11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb41bf1b34464010a1aba0b8438453a2 in term 1.
I20260929 19:03:48.758445 19395 leader_election.cc:302] T 0f8e652b24374d26a6edb03695c61feb P fb41bf1b34464010a1aba0b8438453a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e6d436328547f18c8716ad7fbe1d11, fb41bf1b34464010a1aba0b8438453a2; no voters:
I20260929 19:03:48.758580 19529 raft_consensus.cc:2781] T 0f8e652b24374d26a6edb03695c61feb P fb41bf1b34464010a1aba0b8438453a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:03:48.758720 19529 raft_consensus.cc:686] T 0f8e652b24374d26a6edb03695c61feb P fb41bf1b34464010a1aba0b8438453a2 [term 1 LEADER]: Becoming Leader. State: Replica: fb41bf1b34464010a1aba0b8438453a2, State: Running, Role: LEADER
I20260929 19:03:48.758810 19529 consensus_queue.cc:237] T 0f8e652b24374d26a6edb03695c61feb P fb41bf1b34464010a1aba0b8438453a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11730347cf9f4df3a1fe83c40cef0081" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 37191 } } peers { permanent_uuid: "61e6d436328547f18c8716ad7fbe1d11" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 32937 } } peers { permanent_uuid: "fb41bf1b34464010a1aba0b8438453a2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45345 } }
I20260929 19:03:48.759614 19049 catalog_manager.cc:5495] T 0f8e652b24374d26a6edb03695c61feb P fb41bf1b34464010a1aba0b8438453a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb41bf1b34464010a1aba0b8438453a2 (127.8.158.67). New cstate: current_term: 1 leader_uuid: "fb41bf1b34464010a1aba0b8438453a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11730347cf9f4df3a1fe83c40cef0081" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 37191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61e6d436328547f18c8716ad7fbe1d11" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 32937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb41bf1b34464010a1aba0b8438453a2" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45345 } health_report { overall_health: HEALTHY } } }
I20260929 19:03:48.772868 19521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:48.770452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47316 (local address 127.8.158.67:45345)
0929 19:03:48.770498 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:48.770500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:48.770508 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:03:48.770625 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:48.770625 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:48.770641 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:48.770719 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:03:48.770862 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:48.771335 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:48.772095 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:48.772292 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:48.772367 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:48.772537 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:03:48.772631 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:03:48.772725 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:48.772743 (+ 18us) server_negotiation.cc:299] Negotiation successful
0929 19:03:48.772762 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 19:03:48.774667 19194 raft_consensus.cc:1261] T 0f8e652b24374d26a6edb03695c61feb P 61e6d436328547f18c8716ad7fbe1d11 [term 1 FOLLOWER]: Refusing update from remote peer fb41bf1b34464010a1aba0b8438453a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:03:48.774688 19328 raft_consensus.cc:1261] T 0f8e652b24374d26a6edb03695c61feb P 11730347cf9f4df3a1fe83c40cef0081 [term 1 FOLLOWER]: Refusing update from remote peer fb41bf1b34464010a1aba0b8438453a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:03:48.775023 19529 consensus_queue.cc:1035] T 0f8e652b24374d26a6edb03695c61feb P fb41bf1b34464010a1aba0b8438453a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e6d436328547f18c8716ad7fbe1d11" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 32937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:03:48.775172 19532 consensus_queue.cc:1035] T 0f8e652b24374d26a6edb03695c61feb P fb41bf1b34464010a1aba0b8438453a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11730347cf9f4df3a1fe83c40cef0081" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 37191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:03:48.779476 19048 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:03:48.779580 19048 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:03:48.780287 19048 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 71b593aaeb47461e9948ef76966eeb68: Sending DeleteTablet for 3 replicas of tablet 0f8e652b24374d26a6edb03695c61feb
I20260929 19:03:48.780682 19308 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f8e652b24374d26a6edb03695c61feb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:48 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:46574
I20260929 19:03:48.780697 19174 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f8e652b24374d26a6edb03695c61feb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:48 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:43964
I20260929 19:03:48.780800 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 19111
I20260929 19:03:48.780915 19545 tablet_replica.cc:330] T 0f8e652b24374d26a6edb03695c61feb P 11730347cf9f4df3a1fe83c40cef0081: stopping tablet replica
I20260929 19:03:48.780994 19545 raft_consensus.cc:2229] T 0f8e652b24374d26a6edb03695c61feb P 11730347cf9f4df3a1fe83c40cef0081 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:03:48.781076 19545 raft_consensus.cc:2258] T 0f8e652b24374d26a6edb03695c61feb P 11730347cf9f4df3a1fe83c40cef0081 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:03:48.781272 19442 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f8e652b24374d26a6edb03695c61feb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:48 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:47310
I20260929 19:03:48.781450 19547 tablet_replica.cc:330] T 0f8e652b24374d26a6edb03695c61feb P fb41bf1b34464010a1aba0b8438453a2: stopping tablet replica
I20260929 19:03:48.781513 19547 raft_consensus.cc:2229] T 0f8e652b24374d26a6edb03695c61feb P fb41bf1b34464010a1aba0b8438453a2 [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:03:48.781602 19547 raft_consensus.cc:2258] T 0f8e652b24374d26a6edb03695c61feb P fb41bf1b34464010a1aba0b8438453a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:03:48.782851 19547 ts_tablet_manager.cc:1905] T 0f8e652b24374d26a6edb03695c61feb P fb41bf1b34464010a1aba0b8438453a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:03:48.783959 19545 ts_tablet_manager.cc:1905] T 0f8e652b24374d26a6edb03695c61feb P 11730347cf9f4df3a1fe83c40cef0081: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:03:48.784725 19547 ts_tablet_manager.cc:1918] T 0f8e652b24374d26a6edb03695c61feb P fb41bf1b34464010a1aba0b8438453a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:03:48.784788 19547 log.cc:1196] T 0f8e652b24374d26a6edb03695c61feb P fb41bf1b34464010a1aba0b8438453a2: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-2/wal/wals/0f8e652b24374d26a6edb03695c61feb
I20260929 19:03:48.785049 19547 ts_tablet_manager.cc:1939] T 0f8e652b24374d26a6edb03695c61feb P fb41bf1b34464010a1aba0b8438453a2: Deleting consensus metadata
I20260929 19:03:48.785495 19033 catalog_manager.cc:4841] TS fb41bf1b34464010a1aba0b8438453a2 (127.8.158.67:45345): tablet 0f8e652b24374d26a6edb03695c61feb (table test-table [id=76d98b25b0a14378834cdc2325c5a97e]) successfully deleted
I20260929 19:03:48.785617 19545 ts_tablet_manager.cc:1918] T 0f8e652b24374d26a6edb03695c61feb P 11730347cf9f4df3a1fe83c40cef0081: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:03:48.785678 19545 log.cc:1196] T 0f8e652b24374d26a6edb03695c61feb P 11730347cf9f4df3a1fe83c40cef0081: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708573736807-8825-0/minicluster-data/ts-1/wal/wals/0f8e652b24374d26a6edb03695c61feb
I20260929 19:03:48.785934 19545 ts_tablet_manager.cc:1939] T 0f8e652b24374d26a6edb03695c61feb P 11730347cf9f4df3a1fe83c40cef0081: Deleting consensus metadata
I20260929 19:03:48.786340 19033 catalog_manager.cc:4841] TS 11730347cf9f4df3a1fe83c40cef0081 (127.8.158.66:37191): tablet 0f8e652b24374d26a6edb03695c61feb (table test-table [id=76d98b25b0a14378834cdc2325c5a97e]) successfully deleted
W20260929 19:03:48.787247 19033 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.158.65:32937 (error 108)
I20260929 19:03:48.787395 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 19245
I20260929 19:03:48.788187 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:48.788054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:32937 (local address 127.0.0.1:43970)
0929 19:03:48.788119 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:48.788161 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.65:32937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 19:03:48.788287 19033 catalog_manager.cc:4584] TS 61e6d436328547f18c8716ad7fbe1d11 (127.8.158.65:32937): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0f8e652b24374d26a6edb03695c61feb: Network error: Client connection negotiation failed: client connection to 127.8.158.65:32937: connect: Connection refused (error 111)
I20260929 19:03:48.793797 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 19379
I20260929 19:03:48.798640 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 19017
2026-09-29T19:03:48Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1972 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:03:48 dist-test-slave-nmsz krb5kdc[19551](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:03:48 dist-test-slave-nmsz krb5kdc[19551](info): set up 2 sockets
Sep 29 19:03:48 dist-test-slave-nmsz krb5kdc[19551](info): commencing operation
krb5kdc: starting...
W20260929 19:03:49.835310 8825 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Sep 29 19:03:49 dist-test-slave-nmsz krb5kdc[19551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:03:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:03:49Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.8.158.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.8.158.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.8.158.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/krb5kdc/rangeradmin_127.8.158.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.8.158.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/krb5kdc/rangeradmin_127.8.158.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.8.158.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.8.158.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.8.158.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/krb5kdc/rangerlookup_127.8.158.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.8.158.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/krb5kdc/rangerlookup_127.8.158.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.158.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.8.158.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.158.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/krb5kdc/HTTP_127.8.158.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.158.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/krb5kdc/HTTP_127.8.158.84@KRBTEST.COM.keytab.
I20260929 19:03:50.001169 8825 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskkdiMiM/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/postgres -l logfile start
2026-09-29 19:03:52.655 UTC [19581] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 19:03:52.655 UTC [19581] LOG: listening on IPv4 address "127.8.158.84", port 41047
2026-09-29 19:03:52.659 UTC [19581] LOG: listening on Unix socket "/tmp/.s.PGSQL.41047"
2026-09-29 19:03:52.693 UTC [19584] LOG: database system was shut down at 2026-09-29 19:03:51 UTC
2026-09-29 19:03:52.697 UTC [19581] LOG: database system is ready to accept connections
I20260929 19:03:53.663214 8825 mini_postgres.cc:96] Postgres bound to 41047
2026-09-29 19:03:53.669 UTC [19592] FATAL: database "slave" does not exist
127.8.158.84:41047 - accepting connections
I20260929 19:03:53.670023 8825 mini_ranger.cc:161] Starting Ranger...
I20260929 19:03:53.684216 8825 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 19:03:54.179788 8825 mini_ranger.cc:87] Created ranger Postgres database
I20260929 19:03:54.179878 8825 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ranger-admin
2026-09-29 19:03:54,445 [I] DB FLAVOR :POSTGRES
2026-09-29 19:03:54,445 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 19:03:54,445 [I] Checking connection..
2026-09-29 19:03:54,445 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.158.84:41047/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 19:03:54,683 [I] Checking connection passed.
2026-09-29 19:03:54,684 [I] --------- Verifying version history table ---------
2026-09-29 19:03:54,684 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.158.84:41047/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 19:03:54,920 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 19:03:54,921 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.158.84:41047/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 19:03:55,146 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 19:03:55,147 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 19:03:55,147 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.158.84:41047/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 19:03:55.375 UTC [19656] WARNING: there is no transaction in progress
2026-09-29 19:03:55,381 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 19:03:55,382 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.158.84:41047/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 19:03:55,628 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 19:03:55,628 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 19:03:55,628 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.158.84:41047/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 19:03:55,881 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.158.84:41047/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 19:03:56,120 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.158.84:41047/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-nmsz.c.gcp-upstream.internal','N') ;"
2026-09-29 19:03:56,364 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.158.84:41047/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 19:03:56,617 [I] Table x_portal_user does not exist in database ranger
2026-09-29 19:03:56,618 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.158.84:41047/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 19:03:56,858 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 19:03:56,858 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.158.84:41047/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 19:03:57,115 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.158.84:41047/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 19:03:57,342 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 19:03:57,342 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.158.84:41047/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 19:03:58.295 UTC [19764] WARNING: there is no transaction in progress
2026-09-29 19:03:58.312 UTC [19764] WARNING: there is no transaction in progress
2026-09-29 19:03:58.327 UTC [19764] WARNING: there is no transaction in progress
2026-09-29 19:03:58.342 UTC [19764] WARNING: there is no transaction in progress
2026-09-29 19:03:58.361 UTC [19764] WARNING: there is no transaction in progress
2026-09-29 19:03:58.382 UTC [19764] WARNING: there is no transaction in progress
2026-09-29 19:03:58.448 UTC [19764] WARNING: there is no transaction in progress
2026-09-29 19:03:58.455 UTC [19764] WARNING: there is no transaction in progress
2026-09-29 19:03:58.470 UTC [19764] WARNING: there is no transaction in progress
2026-09-29 19:03:58.481 UTC [19764] WARNING: there is no transaction in progress
2026-09-29 19:03:58.490 UTC [19764] WARNING: there is no transaction in progress
2026-09-29 19:03:58.504 UTC [19764] WARNING: there is no transaction in progress
2026-09-29 19:03:58.514 UTC [19764] WARNING: there is no transaction in progress
2026-09-29 19:03:58.521 UTC [19764] WARNING: there is no transaction in progress
2026-09-29 19:03:58.528 UTC [19764] WARNING: there is no transaction in progress
2026-09-29 19:03:58.535 UTC [19764] WARNING: there is no transaction in progress
2026-09-29 19:03:59,273 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 19:03:59,274 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.158.84:41047/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 19:03:59,508 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 19:03:59,509 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.158.84:41047/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 19:03:59,736 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 19:03:59,736 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.158.84:41047/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 19:03:59,984 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 19:03:59,984 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.158.84:41047/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 19:04:00,224 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.158.84:41047/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 19:04:00,457 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.158.84:41047/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-nmsz.c.gcp-upstream.internal';"
2026-09-29 19:04:00,696 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 19:04:00,696 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.158.84:41047/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 19:04:00,924 [I] DB_PATCHES have already been applied
I20260929 19:04:00.930079 8825 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ranger-admin:/tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskkdiMiM/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 19:04:00.930137 8825 mini_ranger.cc:193] Using host: 127.8.158.84
I20260929 19:04:00.933079 8825 mini_ranger.cc:227] Ranger admin URL: http://127.8.158.84:43935
Sep 29, 2026 7:04:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 7:04:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 7:04:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 7:04:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 7:04:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 7:04:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 7:04:01 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 19:04:02.050464 8825 mini_ranger.cc:160] Time spent starting Ranger: real 8.380s user 0.000s sys 0.009s
Sep 29, 2026 7:04:20 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 7:04:20 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 29, 2026 7:04:20 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 7:04:20 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 7:04:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 7:04:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 7:04:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 7:04:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 7:04:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 7:04:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 7:04:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 7:04:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 7:04:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 7:04:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 7:04:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 7:04:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 7:04:20 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 7:04:21 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 19:04:22.277434 8825 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.8.158.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.8.158.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.8.158.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/krb5kdc/rangerkms_127.8.158.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.8.158.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/krb5kdc/rangerkms_127.8.158.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.158.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.8.158.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.158.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/krb5kdc/HTTP_127.8.158.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.158.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/krb5kdc/HTTP_127.8.158.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 19:04:22 dist-test-slave-nmsz krb5kdc[19551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708662, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 19:04:22.368330 8825 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 19:04:22.377820 8825 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 19:04:23.386226 8825 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 19:04:23.386322 8825 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ranger-kms
I20260929 19:04:23.386363 8825 mini_ranger_kms.cc:227] Using postgres at 127.8.158.84:41047
2026-09-29 19:04:23,549 [I] DB FLAVOR :POSTGRES
2026-09-29 19:04:23,550 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 19:04:23,550 [I] Checking connection
2026-09-29 19:04:23,550 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.158.84:41047/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 19:04:23,810 [I] connection success
2026-09-29 19:04:23,810 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 19:04:23,810 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 19:04:23,810 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.158.84:41047/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 19:04:24,038 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 19:04:24,038 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 19:04:24,038 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 19:04:24,038 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.158.84:41047/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 19:04:24,304 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 19:04:24.764653 8825 mini_ranger_kms.cc:310] Created kms service
I20260929 19:04:24.937180 8825 mini_ranger_kms.cc:326] Created kudu user
I20260929 19:04:24.981269 8825 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 19:04:24 dist-test-slave-nmsz krb5kdc[19551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708662, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.8.158.84@KRBTEST.COM
I20260929 19:04:25.392869 8825 mini_ranger_kms.cc:384] Added ranger policy
I20260929 19:04:25.393136 8825 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ranger-kms:/tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 19:04:25.393159 8825 mini_ranger_kms.cc:240] Using host: 127.8.158.84
I20260929 19:04:25.396323 8825 mini_ranger_kms.cc:276] Ranger KMS PID: 19944
I20260929 19:04:25.396435 8825 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.8.158.84:43147
Sep 29, 2026 7:04:26 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 7:04:26 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 7:04:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 7:04:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 7:04:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 7:04:26 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 19:04:27.674463 8825 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.306s user 0.009s sys 0.136s
I20260929 19:04:27.674577 8825 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 19:04:27.674691 8825 mini_ranger_kms.cc:401] 127.8.158.84:43147/kms/v1/keys
Sep 29 19:04:27 dist-test-slave-nmsz krb5kdc[19551](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790708667, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.158.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 19:04:28 dist-test-slave-nmsz krb5kdc[19551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708667, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.158.84@KRBTEST.COM for HTTP/127.8.158.84@KRBTEST.COM
Sep 29 19:04:28 dist-test-slave-nmsz krb5kdc[19551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708667, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.158.84@KRBTEST.COM for HTTP/127.8.158.84@KRBTEST.COM
Sep 29 19:04:30 dist-test-slave-nmsz krb5kdc[19551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.8.158.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
I20260929 19:04:30.778247 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:46347
--webserver_interface=127.8.158.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:37275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:46347
--ranger_config_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.158.84:43147/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:04:30.865669 19995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:30.865899 19995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:30.865935 19995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:30.867288 19995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:04:30.867357 19995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:30.867383 19995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:30.867408 19995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:04:30.867439 19995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:04:30.868945 19995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:37275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/master-0/wal
--ranger_kms_url=127.8.158.84:43147/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:46347
--ranger_config_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:46347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.158.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:04:30.869212 19995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:30.869431 19995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:04:30.875329 20001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:30.875494 19995 server_base.cc:1034] running on GCE node
W20260929 19:04:30.875349 20003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:30.875725 20000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:30.875975 19995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:30.876267 19995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:30.877436 19995 hybrid_clock.cc:648] HybridClock initialized: now 1790708670877413 us; error 41 us; skew 500 ppm
Sep 29 19:04:30 dist-test-slave-nmsz krb5kdc[19551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708670, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:04:30.892238 19995 init.cc:377] Logged in from keytab as kudu/127.8.158.126@KRBTEST.COM (short username kudu)
I20260929 19:04:30.892741 19995 webserver.cc:460] Webserver started at http://127.8.158.126:41629/ using document root <none> and password file <none>
I20260929 19:04:30.893023 19995 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:30.893087 19995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:30.893221 19995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 19:04:30 dist-test-slave-nmsz krb5kdc[19551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708670, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for HTTP/127.8.158.84@KRBTEST.COM
I20260929 19:04:30.946132 19995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "8bb20225befd436cad6c1890cde434ca"
format_stamp: "Formatted at 2026-09-29 19:04:30 on dist-test-slave-nmsz"
server_key: "cc56817d970d511af27af6ef85bdde49"
server_key_iv: "eefa701a76671858f4a08a154e5cdfa7"
server_key_version: "kuduclusterkey@0"
I20260929 19:04:30.946650 19995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "8bb20225befd436cad6c1890cde434ca"
format_stamp: "Formatted at 2026-09-29 19:04:30 on dist-test-slave-nmsz"
server_key: "cc56817d970d511af27af6ef85bdde49"
server_key_iv: "eefa701a76671858f4a08a154e5cdfa7"
server_key_version: "kuduclusterkey@0"
I20260929 19:04:30.948465 19995 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 19:04:30.962792 20010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:30.963100 19995 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:04:30.963218 19995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "8bb20225befd436cad6c1890cde434ca"
format_stamp: "Formatted at 2026-09-29 19:04:30 on dist-test-slave-nmsz"
server_key: "cc56817d970d511af27af6ef85bdde49"
server_key_iv: "eefa701a76671858f4a08a154e5cdfa7"
server_key_version: "kuduclusterkey@0"
I20260929 19:04:30.963325 19995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:04:30.974766 19995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:30.976687 19995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:30.976904 19995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:30.981948 19995 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.126:46347
I20260929 19:04:30.981998 20062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.126:46347 every 8 connection(s)
I20260929 19:04:30.982424 19995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:04:30.983201 20063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:04:30.984416 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 19995
I20260929 19:04:30.984546 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:04:30.984748 8825 external_mini_cluster.cc:1428] Setting key e67cab57bd277b30d850dcc5af97f463
I20260929 19:04:30.986088 20063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bb20225befd436cad6c1890cde434ca: Bootstrap starting.
I20260929 19:04:30.986866 20063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bb20225befd436cad6c1890cde434ca: Neither blocks nor log segments found. Creating new log.
I20260929 19:04:30.987183 20063 log.cc:824] T 00000000000000000000000000000000 P 8bb20225befd436cad6c1890cde434ca: Log is configured to *not* fsync() on all Append() calls
I20260929 19:04:30.987952 20063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bb20225befd436cad6c1890cde434ca: No bootstrap required, opened a new log
I20260929 19:04:30.990401 20063 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bb20225befd436cad6c1890cde434ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bb20225befd436cad6c1890cde434ca" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 46347 } }
I20260929 19:04:30.990568 20063 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8bb20225befd436cad6c1890cde434ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:04:30.990600 20063 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bb20225befd436cad6c1890cde434ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bb20225befd436cad6c1890cde434ca, State: Initialized, Role: FOLLOWER
I20260929 19:04:30.990767 20063 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bb20225befd436cad6c1890cde434ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bb20225befd436cad6c1890cde434ca" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 46347 } }
I20260929 19:04:30.990861 20063 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8bb20225befd436cad6c1890cde434ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:04:30.990912 20063 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8bb20225befd436cad6c1890cde434ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:04:30.990959 20063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bb20225befd436cad6c1890cde434ca [term 0 FOLLOWER]: Advancing to term 1
Sep 29 19:04:30 dist-test-slave-nmsz krb5kdc[19551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:04:30.991613 20063 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8bb20225befd436cad6c1890cde434ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bb20225befd436cad6c1890cde434ca" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 46347 } }
I20260929 19:04:30.991717 20063 leader_election.cc:302] T 00000000000000000000000000000000 P 8bb20225befd436cad6c1890cde434ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8bb20225befd436cad6c1890cde434ca; no voters:
I20260929 19:04:30.991912 20063 leader_election.cc:288] T 00000000000000000000000000000000 P 8bb20225befd436cad6c1890cde434ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:04:30.991947 20068 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8bb20225befd436cad6c1890cde434ca [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:04:30.992136 20068 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8bb20225befd436cad6c1890cde434ca [term 1 LEADER]: Becoming Leader. State: Replica: 8bb20225befd436cad6c1890cde434ca, State: Running, Role: LEADER
I20260929 19:04:30.992204 20063 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bb20225befd436cad6c1890cde434ca [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:04:30.992272 20068 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bb20225befd436cad6c1890cde434ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bb20225befd436cad6c1890cde434ca" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 46347 } }
I20260929 19:04:30.992702 20068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bb20225befd436cad6c1890cde434ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8bb20225befd436cad6c1890cde434ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bb20225befd436cad6c1890cde434ca" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 46347 } } }
I20260929 19:04:30.992843 20068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bb20225befd436cad6c1890cde434ca [sys.catalog]: This master's current role is: LEADER
I20260929 19:04:30.993060 20068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bb20225befd436cad6c1890cde434ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bb20225befd436cad6c1890cde434ca. Latest consensus state: current_term: 1 leader_uuid: "8bb20225befd436cad6c1890cde434ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bb20225befd436cad6c1890cde434ca" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 46347 } } }
I20260929 19:04:30.993193 20068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bb20225befd436cad6c1890cde434ca [sys.catalog]: This master's current role is: LEADER
I20260929 19:04:30.993361 20077 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:04:30.993631 20063 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 19:04:30.993865 20077 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:04:30.995553 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:30.985298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44456 (local address 127.8.158.126:46347)
0929 19:04:30.985478 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:30.985491 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:30.985533 (+ 42us) server_negotiation.cc:407] Connection header received
0929 19:04:30.985939 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:30.985944 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:30.985989 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:30.986150 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:30.986850 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:30.988023 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:30.988788 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:30.989206 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:30.989326 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:30.991875 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:30.991879 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:30.991879 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:04:30.991890 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:30.993985 (+ 2095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:30.994352 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:30.994355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:30.994356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:30.994450 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:30.994662 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:30.994663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:30.994663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:30.994948 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:30.995091 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:30.995216 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 19:04:30.995326 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":80,"threads_started":1}
I20260929 19:04:31.001889 20085 mvcc.cc:205] Tried to move back new op lower bound from 7334742716398870528 to 7334742716385046528. Current Snapshot: MvccSnapshot[applied={T|T < 7334742716398870528}]
I20260929 19:04:31.004019 20077 catalog_manager.cc:1347] Generated new cluster ID: ecbf4bf1f1b24de69dad15d5a6b95b60
I20260929 19:04:31.004084 20077 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:04:31.012339 20077 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:04:31.013199 20077 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:04:31.019300 20077 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8bb20225befd436cad6c1890cde434ca: Generated new TSK 0
I20260929 19:04:31.019526 20077 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.158.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab.
Entry for principal kudu/127.8.158.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab.
I20260929 19:04:31.749831 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.158.84:43147/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.65:0
--local_ip_for_outbound_sockets=127.8.158.65
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:46347
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab
--principal=kudu/127.8.158.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:37275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:04:31.834940 20108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:31.835155 20108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:31.835193 20108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:31.836359 20108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:31.836431 20108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:31.836474 20108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.65
I20260929 19:04:31.837903 20108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:37275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.8.158.84:43147/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab
--principal=kudu/127.8.158.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:46347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.65
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:04:31.838155 20108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:31.838469 20108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:04:31.843197 20116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:31.843209 20113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:31.843290 20114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:31.843657 20108 server_base.cc:1034] running on GCE node
I20260929 19:04:31.843847 20108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:31.844116 20108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:31.845306 20108 hybrid_clock.cc:648] HybridClock initialized: now 1790708671845285 us; error 36 us; skew 500 ppm
Sep 29 19:04:31 dist-test-slave-nmsz krb5kdc[19551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708671, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:04:31.847885 20108 init.cc:377] Logged in from keytab as kudu/127.8.158.65@KRBTEST.COM (short username kudu)
I20260929 19:04:31.848376 20108 webserver.cc:460] Webserver started at http://127.8.158.65:33487/ using document root <none> and password file <none>
I20260929 19:04:31.848609 20108 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:31.848652 20108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:31.848788 20108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 19:04:31 dist-test-slave-nmsz krb5kdc[19551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708671, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for HTTP/127.8.158.84@KRBTEST.COM
I20260929 19:04:31.862025 20108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-0/data/instance:
uuid: "bcc93c2d6b06433c8955ead5f17cd94d"
format_stamp: "Formatted at 2026-09-29 19:04:31 on dist-test-slave-nmsz"
server_key: "672569c2f332121591fb3bc44d9ac10c"
server_key_iv: "ca5f489decab0fa80f687fc35ba9cdf8"
server_key_version: "kuduclusterkey@0"
I20260929 19:04:31.862514 20108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance:
uuid: "bcc93c2d6b06433c8955ead5f17cd94d"
format_stamp: "Formatted at 2026-09-29 19:04:31 on dist-test-slave-nmsz"
server_key: "672569c2f332121591fb3bc44d9ac10c"
server_key_iv: "ca5f489decab0fa80f687fc35ba9cdf8"
server_key_version: "kuduclusterkey@0"
I20260929 19:04:31.865622 20108 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260929 19:04:31.877396 20123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:31.877611 20108 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:04:31.877707 20108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-0/wal
uuid: "bcc93c2d6b06433c8955ead5f17cd94d"
format_stamp: "Formatted at 2026-09-29 19:04:31 on dist-test-slave-nmsz"
server_key: "672569c2f332121591fb3bc44d9ac10c"
server_key_iv: "ca5f489decab0fa80f687fc35ba9cdf8"
server_key_version: "kuduclusterkey@0"
I20260929 19:04:31.877820 20108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:04:31.914610 20108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:31.917125 20108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:31.917296 20108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:31.917537 20108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:04:31.917886 20108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:04:31.917936 20108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:31.917968 20108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:04:31.918022 20108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:31.925694 20108 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.65:39399
I20260929 19:04:31.926092 20108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
I20260929 19:04:31.928448 20236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.65:39399 every 8 connection(s)
I20260929 19:04:31.934988 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 20108
I20260929 19:04:31.935088 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance
I20260929 19:04:31.935304 8825 external_mini_cluster.cc:1428] Setting key 4d0f43e8d918383fbbd111ee67b0eb26
Sep 29 19:04:31 dist-test-slave-nmsz krb5kdc[19551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708671, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:04:31.939813 20240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:31.929337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:53443 (local address 127.8.158.126:46347)
0929 19:04:31.930572 (+ 1235us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:31.930575 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:31.930583 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:04:31.930609 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:31.930610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:31.930623 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:31.930694 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:31.931484 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:31.932181 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:31.933279 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:31.933613 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:31.933663 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:31.936335 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:31.936336 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:31.936337 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:04:31.936344 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:31.938106 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:31.938680 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:31.938681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:31.938682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:31.938748 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:31.939050 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:31.939051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:31.939052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:31.939196 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:31.939267 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:31.939631 (+ 364us) server_negotiation.cc:299] Negotiation successful
0929 19:04:31.939716 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1198,"thread_start_us":93,"threads_started":1}
I20260929 19:04:31.940299 20239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:31.929051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:46347 (local address 127.8.158.65:53443)
0929 19:04:31.929670 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:31.929718 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:31.930365 (+ 647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:31.930827 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:31.930832 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:31.931080 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:04:31.931351 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:31.931356 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:31.932283 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:31.932284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:31.933146 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:31.933148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:31.933768 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:31.933769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:31.933941 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:31.934608 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:04:31.934617 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:04:31.936239 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:04:31.938207 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:04:31.938214 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:04:31.938216 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:04:31.938565 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:04:31.938825 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:04:31.938826 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:04:31.938826 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:04:31.938952 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:04:31.939298 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:04:31.939304 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:04:31.939460 (+ 156us) client_negotiation.cc:769] Sending connection context
0929 19:04:31.939920 (+ 460us) client_negotiation.cc:240] Negotiation successful
0929 19:04:31.940085 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":77,"threads_started":1}
I20260929 19:04:31.941002 20237 heartbeater.cc:347] Connected to a master server at 127.8.158.126:46347
I20260929 19:04:31.941089 20237 heartbeater.cc:464] Registering TS with master...
I20260929 19:04:31.941263 20237 heartbeater.cc:511] Master 127.8.158.126:46347 requested a full tablet report, sending...
I20260929 19:04:31.941757 20027 ts_manager.cc:194] Registered new tserver with Master: bcc93c2d6b06433c8955ead5f17cd94d (127.8.158.65:39399)
I20260929 19:04:31.942651 20027 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.65@KRBTEST.COM'} at 127.8.158.65:53443
WARNING: no policy specified for kudu/127.8.158.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab.
Entry for principal kudu/127.8.158.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab.
I20260929 19:04:31.965490 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.158.84:43147/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.66:0
--local_ip_for_outbound_sockets=127.8.158.66
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:46347
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab
--principal=kudu/127.8.158.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:37275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:04:32.086884 20245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:32.087205 20245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:32.087286 20245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:32.088891 20245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:32.088994 20245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:32.089071 20245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.66
I20260929 19:04:32.091774 20245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:37275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.8.158.84:43147/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab
--principal=kudu/127.8.158.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:46347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.66
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:04:32.092367 20245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:32.092733 20245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:04:32.099562 20252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:32.099579 20253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:32.100653 20245 server_base.cc:1034] running on GCE node
W20260929 19:04:32.102468 20255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:32.102807 20245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:32.103081 20245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:32.118541 20245 hybrid_clock.cc:648] HybridClock initialized: now 1790708672118501 us; error 49 us; skew 500 ppm
Sep 29 19:04:32 dist-test-slave-nmsz krb5kdc[19551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708672, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:04:32.121850 20245 init.cc:377] Logged in from keytab as kudu/127.8.158.66@KRBTEST.COM (short username kudu)
I20260929 19:04:32.122418 20245 webserver.cc:460] Webserver started at http://127.8.158.66:33285/ using document root <none> and password file <none>
I20260929 19:04:32.122665 20245 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:32.122715 20245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:32.122848 20245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 19:04:32 dist-test-slave-nmsz krb5kdc[19551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708672, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.66@KRBTEST.COM for HTTP/127.8.158.84@KRBTEST.COM
I20260929 19:04:32.135891 20245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-1/data/instance:
uuid: "21ca8c96af394659abc20807d5824c31"
format_stamp: "Formatted at 2026-09-29 19:04:32 on dist-test-slave-nmsz"
server_key: "d6c25257ec0a8ac4a188b138da7b6373"
server_key_iv: "1b76f03236edddfd286eda10040dd019"
server_key_version: "kuduclusterkey@0"
I20260929 19:04:32.137732 20245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance:
uuid: "21ca8c96af394659abc20807d5824c31"
format_stamp: "Formatted at 2026-09-29 19:04:32 on dist-test-slave-nmsz"
server_key: "d6c25257ec0a8ac4a188b138da7b6373"
server_key_iv: "1b76f03236edddfd286eda10040dd019"
server_key_version: "kuduclusterkey@0"
I20260929 19:04:32.141045 20245 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.001s
I20260929 19:04:32.156293 20262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:32.156499 20245 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:04:32.156595 20245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-1/wal
uuid: "21ca8c96af394659abc20807d5824c31"
format_stamp: "Formatted at 2026-09-29 19:04:32 on dist-test-slave-nmsz"
server_key: "d6c25257ec0a8ac4a188b138da7b6373"
server_key_iv: "1b76f03236edddfd286eda10040dd019"
server_key_version: "kuduclusterkey@0"
I20260929 19:04:32.156685 20245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 29 19:04:32 dist-test-slave-nmsz krb5kdc[19551](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790708672, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:04:32.185518 20245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:32.187932 20245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:32.188134 20245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:32.188364 20245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:04:32.188738 20245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:04:32.188825 20245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:32.188899 20245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:04:32.188956 20245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:32.195916 20245 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.66:38851
I20260929 19:04:32.196368 20245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
I20260929 19:04:32.197247 20375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.66:38851 every 8 connection(s)
I20260929 19:04:32.204021 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 20245
I20260929 19:04:32.204118 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance
I20260929 19:04:32.204321 8825 external_mini_cluster.cc:1428] Setting key fce8787dc620a0ee8ba29b12f0514959
Sep 29 19:04:32 dist-test-slave-nmsz krb5kdc[19551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708672, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.66@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:04:32.209164 20240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:32.198095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:34213 (local address 127.8.158.126:46347)
0929 19:04:32.198143 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:32.198145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:32.198418 (+ 273us) server_negotiation.cc:407] Connection header received
0929 19:04:32.199107 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:32.199109 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:32.199125 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:32.199156 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:32.200093 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.200908 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.202388 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.202705 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.202746 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:32.205843 (+ 3097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:32.205845 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:32.205845 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:04:32.205852 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:32.207670 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:32.208132 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:32.208133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:32.208133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:32.208185 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:32.208451 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:32.208452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:32.208453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:32.208573 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:32.208613 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:32.208974 (+ 361us) server_negotiation.cc:299] Negotiation successful
0929 19:04:32.209061 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 19:04:32.209980 20378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:32.198015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:46347 (local address 127.8.158.66:34213)
0929 19:04:32.198267 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:32.198308 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:32.198960 (+ 652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:32.199334 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:32.199337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:32.199698 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:04:32.199968 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:32.199975 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.201024 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.201024 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:32.202252 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:32.202255 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.202833 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.202834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:32.202966 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:32.203541 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:04:32.203551 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:04:32.205746 (+ 2195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:04:32.207766 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:04:32.207771 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:04:32.207773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:04:32.208043 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:04:32.208252 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:04:32.208253 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:04:32.208253 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:04:32.208367 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:04:32.208658 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:04:32.208664 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:04:32.208810 (+ 146us) client_negotiation.cc:769] Sending connection context
0929 19:04:32.209592 (+ 782us) client_negotiation.cc:240] Negotiation successful
0929 19:04:32.209753 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":82,"threads_started":1}
I20260929 19:04:32.210645 20376 heartbeater.cc:347] Connected to a master server at 127.8.158.126:46347
I20260929 19:04:32.210731 20376 heartbeater.cc:464] Registering TS with master...
I20260929 19:04:32.210903 20376 heartbeater.cc:511] Master 127.8.158.126:46347 requested a full tablet report, sending...
I20260929 19:04:32.211277 20027 ts_manager.cc:194] Registered new tserver with Master: 21ca8c96af394659abc20807d5824c31 (127.8.158.66:38851)
I20260929 19:04:32.211752 20027 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.66@KRBTEST.COM'} at 127.8.158.66:34213
WARNING: no policy specified for kudu/127.8.158.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab.
Entry for principal kudu/127.8.158.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab.
I20260929 19:04:32.241219 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.158.84:43147/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.67:0
--local_ip_for_outbound_sockets=127.8.158.67
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:46347
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab
--principal=kudu/127.8.158.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:37275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:04:32.359577 20381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:32.359906 20381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:32.360001 20381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:32.361665 20381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:32.361768 20381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:32.361881 20381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.67
I20260929 19:04:32.364255 20381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:37275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.8.158.84:43147/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab
--principal=kudu/127.8.158.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:46347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.67
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:04:32.364677 20381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:32.365109 20381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:04:32.371970 20388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:32.372454 20387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:32.373113 20390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:32.373431 20381 server_base.cc:1034] running on GCE node
I20260929 19:04:32.373633 20381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:32.373854 20381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:32.375003 20381 hybrid_clock.cc:648] HybridClock initialized: now 1790708672374982 us; error 33 us; skew 500 ppm
Sep 29 19:04:32 dist-test-slave-nmsz krb5kdc[19551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708672, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:04:32.378090 20381 init.cc:377] Logged in from keytab as kudu/127.8.158.67@KRBTEST.COM (short username kudu)
I20260929 19:04:32.378576 20381 webserver.cc:460] Webserver started at http://127.8.158.67:41067/ using document root <none> and password file <none>
I20260929 19:04:32.378801 20381 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:32.378849 20381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:32.378986 20381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 19:04:32 dist-test-slave-nmsz krb5kdc[19551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708672, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.67@KRBTEST.COM for HTTP/127.8.158.84@KRBTEST.COM
I20260929 19:04:32.397279 20381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-2/data/instance:
uuid: "5666812ae7b6450faa397d40441279fc"
format_stamp: "Formatted at 2026-09-29 19:04:32 on dist-test-slave-nmsz"
server_key: "8364e2bd2c874bd623b1905c8a2f2e08"
server_key_iv: "209e68a0630ffd680dcd08ddfefd363f"
server_key_version: "kuduclusterkey@0"
I20260929 19:04:32.397701 20381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance:
uuid: "5666812ae7b6450faa397d40441279fc"
format_stamp: "Formatted at 2026-09-29 19:04:32 on dist-test-slave-nmsz"
server_key: "8364e2bd2c874bd623b1905c8a2f2e08"
server_key_iv: "209e68a0630ffd680dcd08ddfefd363f"
server_key_version: "kuduclusterkey@0"
I20260929 19:04:32.400275 20381 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 19:04:32.417855 20397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:32.418094 20381 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260929 19:04:32.418172 20381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-2/wal
uuid: "5666812ae7b6450faa397d40441279fc"
format_stamp: "Formatted at 2026-09-29 19:04:32 on dist-test-slave-nmsz"
server_key: "8364e2bd2c874bd623b1905c8a2f2e08"
server_key_iv: "209e68a0630ffd680dcd08ddfefd363f"
server_key_version: "kuduclusterkey@0"
I20260929 19:04:32.418251 20381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:04:32.440343 20381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:32.443013 20381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:32.443154 20381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:32.443347 20381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:04:32.443693 20381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:04:32.443740 20381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:32.443769 20381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:04:32.443835 20381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:32.450034 20381 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.67:38171
I20260929 19:04:32.450518 20381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
I20260929 19:04:32.459018 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 20381
I20260929 19:04:32.459105 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance
I20260929 19:04:32.459295 8825 external_mini_cluster.cc:1428] Setting key a94ec89706ad61fc099bba76a0050422
I20260929 19:04:32.460536 20510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.67:38171 every 8 connection(s)
Sep 29 19:04:32 dist-test-slave-nmsz krb5kdc[19551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708672, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.67@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:04:32.473706 20240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:32.461174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:38187 (local address 127.8.158.126:46347)
0929 19:04:32.461220 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:32.461222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:32.463140 (+ 1918us) server_negotiation.cc:407] Connection header received
0929 19:04:32.463786 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:32.463787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:32.463799 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:32.463829 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:32.464633 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.465544 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.467056 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.467336 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.467378 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:32.470214 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:32.470215 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:32.470215 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:04:32.470223 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:32.472260 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:32.472710 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:32.472710 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:32.472711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:32.472767 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:32.473034 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:32.473035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:32.473035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:32.473157 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:32.473197 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:32.473534 (+ 337us) server_negotiation.cc:299] Negotiation successful
0929 19:04:32.473612 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 19:04:32.474269 20513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:32.461178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:46347 (local address 127.8.158.67:38187)
0929 19:04:32.463010 (+ 1832us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:32.463057 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:32.463668 (+ 611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:32.464020 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:32.464024 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:32.464259 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:04:32.464524 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:32.464529 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.465658 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.465659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:32.466955 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:32.466958 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.467467 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.467468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:32.467650 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:32.468247 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:04:32.468257 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:04:32.470108 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:04:32.472349 (+ 2241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:04:32.472355 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:04:32.472357 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:04:32.472624 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:04:32.472834 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:04:32.472834 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:04:32.472835 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:04:32.472961 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:04:32.473240 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:04:32.473245 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:04:32.473389 (+ 144us) client_negotiation.cc:769] Sending connection context
0929 19:04:32.473890 (+ 501us) client_negotiation.cc:240] Negotiation successful
0929 19:04:32.474051 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1749,"thread_start_us":106,"threads_started":1}
I20260929 19:04:32.474790 20511 heartbeater.cc:347] Connected to a master server at 127.8.158.126:46347
I20260929 19:04:32.474875 20511 heartbeater.cc:464] Registering TS with master...
I20260929 19:04:32.475054 20511 heartbeater.cc:511] Master 127.8.158.126:46347 requested a full tablet report, sending...
I20260929 19:04:32.475422 20027 ts_manager.cc:194] Registered new tserver with Master: 5666812ae7b6450faa397d40441279fc (127.8.158.67:38171)
I20260929 19:04:32.475870 20027 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.67@KRBTEST.COM'} at 127.8.158.67:38187
I20260929 19:04:32.481855 8825 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:04:32.494035 20240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:32.483418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44468 (local address 127.8.158.126:46347)
0929 19:04:32.483455 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:32.483457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:32.486535 (+ 3078us) server_negotiation.cc:407] Connection header received
0929 19:04:32.486591 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:32.486592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:32.486605 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:32.486660 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:32.487351 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.488091 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.489132 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.489398 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.489467 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:32.490905 (+ 1438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:32.490907 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:32.490907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:04:32.490915 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:32.492769 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:32.493167 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:32.493168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:32.493169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:32.493240 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:32.493455 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:32.493456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:32.493456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:32.493603 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:32.493715 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:32.493824 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 19:04:32.493911 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 19:04:32.495846 20027 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44468:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:04:32.498611 20027 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:04:32.511152 20523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:32.503938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:39399 (local address 127.0.0.1:33598)
0929 19:04:32.504728 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:32.504735 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:32.504769 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:32.505383 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:32.505383 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:32.505384 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:04:32.505413 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:32.505414 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.506953 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.506954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:32.509637 (+ 2683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:32.509640 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.510858 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.510859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:32.510950 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:32.510962 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 19:04:32.511024 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 19:04:32.511079 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"thread_start_us":56,"threads_started":1}
I20260929 19:04:32.511178 20524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:32.503594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59088 (local address 127.8.158.66:38851)
0929 19:04:32.505062 (+ 1468us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:32.505066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:32.505074 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:04:32.505110 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:32.505112 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:32.505125 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:32.505186 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:04:32.507766 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.508793 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.510059 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.510751 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.510816 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:32.510884 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:32.511067 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 19:04:32.511100 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1433,"thread_start_us":69,"threads_started":1}
I20260929 19:04:32.511853 20527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:32.504181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33598 (local address 127.8.158.65:39399)
0929 19:04:32.504381 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:32.504386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:32.504819 (+ 433us) server_negotiation.cc:407] Connection header received
0929 19:04:32.504855 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:32.504858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:32.504873 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:32.504950 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:04:32.505478 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.506853 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.509729 (+ 2876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.510709 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.510776 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:32.510827 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:32.511746 (+ 919us) server_negotiation.cc:299] Negotiation successful
0929 19:04:32.511784 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":45,"threads_started":1}
I20260929 19:04:32.511686 20526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:32.503355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.66:38851 (local address 127.0.0.1:59088)
0929 19:04:32.504444 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:32.504481 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:32.504541 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:32.505259 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:32.505262 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:32.505264 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:04:32.505320 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:32.505321 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.508889 (+ 3568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.508890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:32.509936 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:32.509938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.510917 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.510918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:32.510976 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:32.510980 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:04:32.511027 (+ 47us) client_negotiation.cc:240] Negotiation successful
0929 19:04:32.511625 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1062,"mutex_wait_us":563,"thread_start_us":70,"threads_started":1}
I20260929 19:04:32.512645 20171 tablet_service.cc:1467] Processing CreateTablet for tablet 62ccbd4b99764f078781c3b8d11adff9 (DEFAULT_TABLE table=test-table [id=fd20f80630a44daab373b8ad4bbbbbc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:04:32.512992 20171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62ccbd4b99764f078781c3b8d11adff9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:04:32.512950 20310 tablet_service.cc:1467] Processing CreateTablet for tablet 62ccbd4b99764f078781c3b8d11adff9 (DEFAULT_TABLE table=test-table [id=fd20f80630a44daab373b8ad4bbbbbc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:04:32.513463 20310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62ccbd4b99764f078781c3b8d11adff9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:04:32.517246 20530 tablet_bootstrap.cc:492] T 62ccbd4b99764f078781c3b8d11adff9 P 21ca8c96af394659abc20807d5824c31: Bootstrap starting.
I20260929 19:04:32.518287 20530 tablet_bootstrap.cc:654] T 62ccbd4b99764f078781c3b8d11adff9 P 21ca8c96af394659abc20807d5824c31: Neither blocks nor log segments found. Creating new log.
I20260929 19:04:32.518666 20530 log.cc:824] T 62ccbd4b99764f078781c3b8d11adff9 P 21ca8c96af394659abc20807d5824c31: Log is configured to *not* fsync() on all Append() calls
I20260929 19:04:32.519384 20530 tablet_bootstrap.cc:492] T 62ccbd4b99764f078781c3b8d11adff9 P 21ca8c96af394659abc20807d5824c31: No bootstrap required, opened a new log
I20260929 19:04:32.519445 20530 ts_tablet_manager.cc:1397] T 62ccbd4b99764f078781c3b8d11adff9 P 21ca8c96af394659abc20807d5824c31: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:04:32.521242 20530 raft_consensus.cc:348] T 62ccbd4b99764f078781c3b8d11adff9 P 21ca8c96af394659abc20807d5824c31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ca8c96af394659abc20807d5824c31" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38851 } } peers { permanent_uuid: "5666812ae7b6450faa397d40441279fc" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38171 } } peers { permanent_uuid: "bcc93c2d6b06433c8955ead5f17cd94d" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 39399 } }
I20260929 19:04:32.521384 20530 raft_consensus.cc:374] T 62ccbd4b99764f078781c3b8d11adff9 P 21ca8c96af394659abc20807d5824c31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:04:32.521414 20530 raft_consensus.cc:729] T 62ccbd4b99764f078781c3b8d11adff9 P 21ca8c96af394659abc20807d5824c31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21ca8c96af394659abc20807d5824c31, State: Initialized, Role: FOLLOWER
I20260929 19:04:32.521516 20530 consensus_queue.cc:260] T 62ccbd4b99764f078781c3b8d11adff9 P 21ca8c96af394659abc20807d5824c31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ca8c96af394659abc20807d5824c31" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38851 } } peers { permanent_uuid: "5666812ae7b6450faa397d40441279fc" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38171 } } peers { permanent_uuid: "bcc93c2d6b06433c8955ead5f17cd94d" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 39399 } }
I20260929 19:04:32.521725 20530 ts_tablet_manager.cc:1428] T 62ccbd4b99764f078781c3b8d11adff9 P 21ca8c96af394659abc20807d5824c31: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:04:32.521939 20376 heartbeater.cc:503] Master 127.8.158.126:46347 was elected leader, sending a full tablet report...
I20260929 19:04:32.522456 20529 tablet_bootstrap.cc:492] T 62ccbd4b99764f078781c3b8d11adff9 P bcc93c2d6b06433c8955ead5f17cd94d: Bootstrap starting.
I20260929 19:04:32.523115 20529 tablet_bootstrap.cc:654] T 62ccbd4b99764f078781c3b8d11adff9 P bcc93c2d6b06433c8955ead5f17cd94d: Neither blocks nor log segments found. Creating new log.
I20260929 19:04:32.523497 20529 log.cc:824] T 62ccbd4b99764f078781c3b8d11adff9 P bcc93c2d6b06433c8955ead5f17cd94d: Log is configured to *not* fsync() on all Append() calls
I20260929 19:04:32.527197 20529 tablet_bootstrap.cc:492] T 62ccbd4b99764f078781c3b8d11adff9 P bcc93c2d6b06433c8955ead5f17cd94d: No bootstrap required, opened a new log
I20260929 19:04:32.527302 20529 ts_tablet_manager.cc:1397] T 62ccbd4b99764f078781c3b8d11adff9 P bcc93c2d6b06433c8955ead5f17cd94d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260929 19:04:32.529651 20529 raft_consensus.cc:348] T 62ccbd4b99764f078781c3b8d11adff9 P bcc93c2d6b06433c8955ead5f17cd94d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ca8c96af394659abc20807d5824c31" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38851 } } peers { permanent_uuid: "5666812ae7b6450faa397d40441279fc" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38171 } } peers { permanent_uuid: "bcc93c2d6b06433c8955ead5f17cd94d" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 39399 } }
I20260929 19:04:32.529839 20529 raft_consensus.cc:374] T 62ccbd4b99764f078781c3b8d11adff9 P bcc93c2d6b06433c8955ead5f17cd94d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:04:32.529917 20529 raft_consensus.cc:729] T 62ccbd4b99764f078781c3b8d11adff9 P bcc93c2d6b06433c8955ead5f17cd94d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcc93c2d6b06433c8955ead5f17cd94d, State: Initialized, Role: FOLLOWER
I20260929 19:04:32.530057 20529 consensus_queue.cc:260] T 62ccbd4b99764f078781c3b8d11adff9 P bcc93c2d6b06433c8955ead5f17cd94d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ca8c96af394659abc20807d5824c31" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38851 } } peers { permanent_uuid: "5666812ae7b6450faa397d40441279fc" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38171 } } peers { permanent_uuid: "bcc93c2d6b06433c8955ead5f17cd94d" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 39399 } }
I20260929 19:04:32.530342 20529 ts_tablet_manager.cc:1428] T 62ccbd4b99764f078781c3b8d11adff9 P bcc93c2d6b06433c8955ead5f17cd94d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 19:04:32.530496 20237 heartbeater.cc:503] Master 127.8.158.126:46347 was elected leader, sending a full tablet report...
I20260929 19:04:32.531795 20525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:32.503794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.67:38171 (local address 127.0.0.1:58394)
0929 19:04:32.504629 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:32.504637 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:32.504667 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:32.516528 (+ 11861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:32.516529 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:32.516531 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:04:32.516565 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:32.516566 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.517776 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.517777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:32.530446 (+ 12669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:32.530449 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.531543 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.531544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:32.531615 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:32.531620 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 19:04:32.531669 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 19:04:32.531715 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"thread_start_us":61,"threads_started":1}
I20260929 19:04:32.532294 20528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:32.512669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58394 (local address 127.8.158.67:38171)
0929 19:04:32.516280 (+ 3611us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:32.516285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:32.516297 (+ 12us) server_negotiation.cc:407] Connection header received
0929 19:04:32.516334 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:32.516336 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:32.516351 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:32.516448 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:04:32.516647 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.517675 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.530591 (+ 12916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.531375 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.531440 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:32.531510 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:32.532158 (+ 648us) server_negotiation.cc:299] Negotiation successful
0929 19:04:32.532206 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3575,"thread_start_us":80,"threads_started":1}
I20260929 19:04:32.533033 20444 tablet_service.cc:1467] Processing CreateTablet for tablet 62ccbd4b99764f078781c3b8d11adff9 (DEFAULT_TABLE table=test-table [id=fd20f80630a44daab373b8ad4bbbbbc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:04:32.533344 20444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62ccbd4b99764f078781c3b8d11adff9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:04:32.540470 20535 tablet_bootstrap.cc:492] T 62ccbd4b99764f078781c3b8d11adff9 P 5666812ae7b6450faa397d40441279fc: Bootstrap starting.
I20260929 19:04:32.542842 20535 tablet_bootstrap.cc:654] T 62ccbd4b99764f078781c3b8d11adff9 P 5666812ae7b6450faa397d40441279fc: Neither blocks nor log segments found. Creating new log.
I20260929 19:04:32.543188 20535 log.cc:824] T 62ccbd4b99764f078781c3b8d11adff9 P 5666812ae7b6450faa397d40441279fc: Log is configured to *not* fsync() on all Append() calls
I20260929 19:04:32.544090 20535 tablet_bootstrap.cc:492] T 62ccbd4b99764f078781c3b8d11adff9 P 5666812ae7b6450faa397d40441279fc: No bootstrap required, opened a new log
I20260929 19:04:32.544157 20535 ts_tablet_manager.cc:1397] T 62ccbd4b99764f078781c3b8d11adff9 P 5666812ae7b6450faa397d40441279fc: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260929 19:04:32.546092 20535 raft_consensus.cc:348] T 62ccbd4b99764f078781c3b8d11adff9 P 5666812ae7b6450faa397d40441279fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ca8c96af394659abc20807d5824c31" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38851 } } peers { permanent_uuid: "5666812ae7b6450faa397d40441279fc" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38171 } } peers { permanent_uuid: "bcc93c2d6b06433c8955ead5f17cd94d" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 39399 } }
I20260929 19:04:32.546239 20535 raft_consensus.cc:374] T 62ccbd4b99764f078781c3b8d11adff9 P 5666812ae7b6450faa397d40441279fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:04:32.546268 20535 raft_consensus.cc:729] T 62ccbd4b99764f078781c3b8d11adff9 P 5666812ae7b6450faa397d40441279fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5666812ae7b6450faa397d40441279fc, State: Initialized, Role: FOLLOWER
I20260929 19:04:32.549994 20535 consensus_queue.cc:260] T 62ccbd4b99764f078781c3b8d11adff9 P 5666812ae7b6450faa397d40441279fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ca8c96af394659abc20807d5824c31" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38851 } } peers { permanent_uuid: "5666812ae7b6450faa397d40441279fc" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38171 } } peers { permanent_uuid: "bcc93c2d6b06433c8955ead5f17cd94d" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 39399 } }
I20260929 19:04:32.550251 20535 ts_tablet_manager.cc:1428] T 62ccbd4b99764f078781c3b8d11adff9 P 5666812ae7b6450faa397d40441279fc: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260929 19:04:32.550382 20511 heartbeater.cc:503] Master 127.8.158.126:46347 was elected leader, sending a full tablet report...
I20260929 19:04:32.627620 20537 raft_consensus.cc:482] T 62ccbd4b99764f078781c3b8d11adff9 P 5666812ae7b6450faa397d40441279fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:04:32.627766 20537 raft_consensus.cc:504] T 62ccbd4b99764f078781c3b8d11adff9 P 5666812ae7b6450faa397d40441279fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ca8c96af394659abc20807d5824c31" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38851 } } peers { permanent_uuid: "5666812ae7b6450faa397d40441279fc" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38171 } } peers { permanent_uuid: "bcc93c2d6b06433c8955ead5f17cd94d" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 39399 } }
I20260929 19:04:32.628173 20537 leader_election.cc:288] T 62ccbd4b99764f078781c3b8d11adff9 P 5666812ae7b6450faa397d40441279fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21ca8c96af394659abc20807d5824c31 (127.8.158.66:38851), bcc93c2d6b06433c8955ead5f17cd94d (127.8.158.65:39399)
I20260929 19:04:32.632608 20513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:32.628435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:39399 (local address 127.8.158.67:44923)
0929 19:04:32.628628 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:32.628639 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:32.628685 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:32.629298 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:32.629299 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:32.629301 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:04:32.629369 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:32.629370 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.630345 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.630346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:32.631526 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:32.631529 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.632366 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.632367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:32.632443 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:32.632446 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:04:32.632494 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 19:04:32.632532 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172}
I20260929 19:04:32.633292 20527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:32.628763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:44923 (local address 127.8.158.65:39399)
0929 19:04:32.628803 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:32.628805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:32.628812 (+ 7us) server_negotiation.cc:407] Connection header received
0929 19:04:32.628836 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:32.628837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:32.628862 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:32.628907 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:04:32.629454 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.630215 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.631625 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.632251 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.632324 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:32.632334 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:32.633185 (+ 851us) server_negotiation.cc:299] Negotiation successful
0929 19:04:32.633228 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 19:04:32.633543 20191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62ccbd4b99764f078781c3b8d11adff9" candidate_uuid: "5666812ae7b6450faa397d40441279fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcc93c2d6b06433c8955ead5f17cd94d" is_pre_election: true
I20260929 19:04:32.633703 20191 raft_consensus.cc:2454] T 62ccbd4b99764f078781c3b8d11adff9 P bcc93c2d6b06433c8955ead5f17cd94d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5666812ae7b6450faa397d40441279fc in term 0.
I20260929 19:04:32.634636 20398 leader_election.cc:302] T 62ccbd4b99764f078781c3b8d11adff9 P 5666812ae7b6450faa397d40441279fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5666812ae7b6450faa397d40441279fc, bcc93c2d6b06433c8955ead5f17cd94d; no voters:
I20260929 19:04:32.634827 20537 raft_consensus.cc:2781] T 62ccbd4b99764f078781c3b8d11adff9 P 5666812ae7b6450faa397d40441279fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:04:32.634886 20537 raft_consensus.cc:482] T 62ccbd4b99764f078781c3b8d11adff9 P 5666812ae7b6450faa397d40441279fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:04:32.634907 20537 raft_consensus.cc:3037] T 62ccbd4b99764f078781c3b8d11adff9 P 5666812ae7b6450faa397d40441279fc [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:04:32.635777 20537 raft_consensus.cc:504] T 62ccbd4b99764f078781c3b8d11adff9 P 5666812ae7b6450faa397d40441279fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ca8c96af394659abc20807d5824c31" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38851 } } peers { permanent_uuid: "5666812ae7b6450faa397d40441279fc" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38171 } } peers { permanent_uuid: "bcc93c2d6b06433c8955ead5f17cd94d" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 39399 } }
I20260929 19:04:32.635923 20537 leader_election.cc:288] T 62ccbd4b99764f078781c3b8d11adff9 P 5666812ae7b6450faa397d40441279fc [CANDIDATE]: Term 1 election: Requested vote from peers 21ca8c96af394659abc20807d5824c31 (127.8.158.66:38851), bcc93c2d6b06433c8955ead5f17cd94d (127.8.158.65:39399)
I20260929 19:04:32.636206 20191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62ccbd4b99764f078781c3b8d11adff9" candidate_uuid: "5666812ae7b6450faa397d40441279fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcc93c2d6b06433c8955ead5f17cd94d"
I20260929 19:04:32.636279 20191 raft_consensus.cc:3037] T 62ccbd4b99764f078781c3b8d11adff9 P bcc93c2d6b06433c8955ead5f17cd94d [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:04:32.637122 20191 raft_consensus.cc:2454] T 62ccbd4b99764f078781c3b8d11adff9 P bcc93c2d6b06433c8955ead5f17cd94d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5666812ae7b6450faa397d40441279fc in term 1.
I20260929 19:04:32.637334 20398 leader_election.cc:302] T 62ccbd4b99764f078781c3b8d11adff9 P 5666812ae7b6450faa397d40441279fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5666812ae7b6450faa397d40441279fc, bcc93c2d6b06433c8955ead5f17cd94d; no voters:
I20260929 19:04:32.637447 20537 raft_consensus.cc:2781] T 62ccbd4b99764f078781c3b8d11adff9 P 5666812ae7b6450faa397d40441279fc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:04:32.637590 20537 raft_consensus.cc:686] T 62ccbd4b99764f078781c3b8d11adff9 P 5666812ae7b6450faa397d40441279fc [term 1 LEADER]: Becoming Leader. State: Replica: 5666812ae7b6450faa397d40441279fc, State: Running, Role: LEADER
I20260929 19:04:32.637671 20537 consensus_queue.cc:237] T 62ccbd4b99764f078781c3b8d11adff9 P 5666812ae7b6450faa397d40441279fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ca8c96af394659abc20807d5824c31" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38851 } } peers { permanent_uuid: "5666812ae7b6450faa397d40441279fc" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38171 } } peers { permanent_uuid: "bcc93c2d6b06433c8955ead5f17cd94d" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 39399 } }
I20260929 19:04:32.638586 20027 catalog_manager.cc:5495] T 62ccbd4b99764f078781c3b8d11adff9 P 5666812ae7b6450faa397d40441279fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5666812ae7b6450faa397d40441279fc (127.8.158.67). New cstate: current_term: 1 leader_uuid: "5666812ae7b6450faa397d40441279fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ca8c96af394659abc20807d5824c31" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5666812ae7b6450faa397d40441279fc" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 38171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcc93c2d6b06433c8955ead5f17cd94d" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 39399 } health_report { overall_health: UNKNOWN } } }
I20260929 19:04:32.641034 20513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:32.629038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.66:38851 (local address 127.8.158.67:46695)
0929 19:04:32.632674 (+ 3636us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:32.632685 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:32.632725 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:32.633600 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:32.633602 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:32.633603 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:04:32.633636 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:32.633637 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.639018 (+ 5381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.639019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:32.640085 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:32.640088 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.640813 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.640814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:32.640882 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:32.640885 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:04:32.640931 (+ 46us) client_negotiation.cc:240] Negotiation successful
0929 19:04:32.640965 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3625,"thread_start_us":92,"threads_started":1}
I20260929 19:04:32.641413 20524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:32.629246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:46695 (local address 127.8.158.66:38851)
0929 19:04:32.629291 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:32.629293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:32.632779 (+ 3486us) server_negotiation.cc:407] Connection header received
0929 19:04:32.632807 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:32.632808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:32.632820 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:32.632876 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:04:32.633722 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.634460 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.640185 (+ 5725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.640706 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.640776 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:32.640784 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:32.641314 (+ 530us) server_negotiation.cc:299] Negotiation successful
0929 19:04:32.641352 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 19:04:32.641942 20316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62ccbd4b99764f078781c3b8d11adff9" candidate_uuid: "5666812ae7b6450faa397d40441279fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21ca8c96af394659abc20807d5824c31" is_pre_election: true
I20260929 19:04:32.642076 20316 raft_consensus.cc:2454] T 62ccbd4b99764f078781c3b8d11adff9 P 21ca8c96af394659abc20807d5824c31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5666812ae7b6450faa397d40441279fc in term 0.
I20260929 19:04:32.642174 20315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62ccbd4b99764f078781c3b8d11adff9" candidate_uuid: "5666812ae7b6450faa397d40441279fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21ca8c96af394659abc20807d5824c31"
I20260929 19:04:32.642225 20315 raft_consensus.cc:3037] T 62ccbd4b99764f078781c3b8d11adff9 P 21ca8c96af394659abc20807d5824c31 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:04:32.643030 20315 raft_consensus.cc:2454] T 62ccbd4b99764f078781c3b8d11adff9 P 21ca8c96af394659abc20807d5824c31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5666812ae7b6450faa397d40441279fc in term 1.
I20260929 19:04:32.659644 20528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:32.656203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58402 (local address 127.8.158.67:38171)
0929 19:04:32.656255 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:32.656258 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:32.656268 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:04:32.656305 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:32.656306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:32.656325 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:32.656412 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:04:32.656619 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.657383 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.658535 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.658841 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.658908 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:32.659093 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:04:32.659214 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:04:32.659282 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:32.659528 (+ 246us) server_negotiation.cc:299] Negotiation successful
0929 19:04:32.659547 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 19:04:32.670200 20191 raft_consensus.cc:1261] T 62ccbd4b99764f078781c3b8d11adff9 P bcc93c2d6b06433c8955ead5f17cd94d [term 1 FOLLOWER]: Refusing update from remote peer 5666812ae7b6450faa397d40441279fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:04:32.670648 20539 consensus_queue.cc:1035] T 62ccbd4b99764f078781c3b8d11adff9 P 5666812ae7b6450faa397d40441279fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcc93c2d6b06433c8955ead5f17cd94d" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 39399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Sep 29 19:04:32 dist-test-slave-nmsz krb5kdc[19551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708672, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for HTTP/127.8.158.84@KRBTEST.COM
I20260929 19:04:32.672796 20315 raft_consensus.cc:1261] T 62ccbd4b99764f078781c3b8d11adff9 P 21ca8c96af394659abc20807d5824c31 [term 1 FOLLOWER]: Refusing update from remote peer 5666812ae7b6450faa397d40441279fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:04:32.673126 20539 consensus_queue.cc:1035] T 62ccbd4b99764f078781c3b8d11adff9 P 5666812ae7b6450faa397d40441279fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "21ca8c96af394659abc20807d5824c31" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 19:04:32.676911 20238 tablet.cc:2369] T 62ccbd4b99764f078781c3b8d11adff9 P bcc93c2d6b06433c8955ead5f17cd94d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:04:32.680645 20542 mvcc.cc:205] Tried to move back new op lower bound from 7334742723246616576 to 7334742723124482048. Current Snapshot: MvccSnapshot[applied={T|T < 7334742723246616576}]
I20260929 19:04:32.680877 20545 mvcc.cc:205] Tried to move back new op lower bound from 7334742723246616576 to 7334742723124482048. Current Snapshot: MvccSnapshot[applied={T|T < 7334742723246616576}]
I20260929 19:04:32.683257 20543 mvcc.cc:205] Tried to move back new op lower bound from 7334742723246616576 to 7334742723124482048. Current Snapshot: MvccSnapshot[applied={T|T < 7334742723246616576}]
I20260929 19:04:32.685516 20027 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:04:32.685606 20027 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:04:32.686301 20027 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8bb20225befd436cad6c1890cde434ca: Sending DeleteTablet for 3 replicas of tablet 62ccbd4b99764f078781c3b8d11adff9
I20260929 19:04:32.686923 20310 tablet_service.cc:1514] Processing DeleteTablet for tablet 62ccbd4b99764f078781c3b8d11adff9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:04:32 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:59088
I20260929 19:04:32.687132 20444 tablet_service.cc:1514] Processing DeleteTablet for tablet 62ccbd4b99764f078781c3b8d11adff9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:04:32 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:58394
I20260929 19:04:32.687309 20171 tablet_service.cc:1514] Processing DeleteTablet for tablet 62ccbd4b99764f078781c3b8d11adff9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:04:32 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:33598
I20260929 19:04:32.687824 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 20108
I20260929 19:04:32.692994 20552 tablet_replica.cc:330] T 62ccbd4b99764f078781c3b8d11adff9 P 5666812ae7b6450faa397d40441279fc: stopping tablet replica
I20260929 19:04:32.693102 20552 raft_consensus.cc:2229] T 62ccbd4b99764f078781c3b8d11adff9 P 5666812ae7b6450faa397d40441279fc [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:04:32.693226 20552 raft_consensus.cc:2258] T 62ccbd4b99764f078781c3b8d11adff9 P 5666812ae7b6450faa397d40441279fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:04:32.693374 20551 tablet_replica.cc:330] T 62ccbd4b99764f078781c3b8d11adff9 P 21ca8c96af394659abc20807d5824c31: stopping tablet replica
I20260929 19:04:32.693467 20551 raft_consensus.cc:2229] T 62ccbd4b99764f078781c3b8d11adff9 P 21ca8c96af394659abc20807d5824c31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:04:32.693559 20551 raft_consensus.cc:2258] T 62ccbd4b99764f078781c3b8d11adff9 P 21ca8c96af394659abc20807d5824c31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:04:32.695716 20551 ts_tablet_manager.cc:1905] T 62ccbd4b99764f078781c3b8d11adff9 P 21ca8c96af394659abc20807d5824c31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:04:32.697306 20551 ts_tablet_manager.cc:1918] T 62ccbd4b99764f078781c3b8d11adff9 P 21ca8c96af394659abc20807d5824c31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:04:32.697383 20551 log.cc:1196] T 62ccbd4b99764f078781c3b8d11adff9 P 21ca8c96af394659abc20807d5824c31: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-1/wal/wals/62ccbd4b99764f078781c3b8d11adff9
I20260929 19:04:32.697652 20551 ts_tablet_manager.cc:1939] T 62ccbd4b99764f078781c3b8d11adff9 P 21ca8c96af394659abc20807d5824c31: Deleting consensus metadata
I20260929 19:04:32.698081 20014 catalog_manager.cc:4841] TS 21ca8c96af394659abc20807d5824c31 (127.8.158.66:38851): tablet 62ccbd4b99764f078781c3b8d11adff9 (table test-table [id=fd20f80630a44daab373b8ad4bbbbbc5]) successfully deleted
I20260929 19:04:32.710752 20552 ts_tablet_manager.cc:1905] T 62ccbd4b99764f078781c3b8d11adff9 P 5666812ae7b6450faa397d40441279fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:04:32.714201 20552 ts_tablet_manager.cc:1918] T 62ccbd4b99764f078781c3b8d11adff9 P 5666812ae7b6450faa397d40441279fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:04:32.714401 20552 log.cc:1196] T 62ccbd4b99764f078781c3b8d11adff9 P 5666812ae7b6450faa397d40441279fc: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708573736807-8825-0/minicluster-data/ts-2/wal/wals/62ccbd4b99764f078781c3b8d11adff9
I20260929 19:04:32.714754 20552 ts_tablet_manager.cc:1939] T 62ccbd4b99764f078781c3b8d11adff9 P 5666812ae7b6450faa397d40441279fc: Deleting consensus metadata
I20260929 19:04:32.715255 20011 catalog_manager.cc:4841] TS 5666812ae7b6450faa397d40441279fc (127.8.158.67:38171): tablet 62ccbd4b99764f078781c3b8d11adff9 (table test-table [id=fd20f80630a44daab373b8ad4bbbbbc5]) successfully deleted
W20260929 19:04:32.715631 20011 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.158.65:39399 (error 108)
I20260929 19:04:32.716493 20525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:32.716376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:39399 (local address 127.0.0.1:33608)
0929 19:04:32.716420 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:32.716458 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.65:39399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260929 19:04:32.716506 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 20245
W20260929 19:04:32.716653 20011 catalog_manager.cc:4584] TS bcc93c2d6b06433c8955ead5f17cd94d (127.8.158.65:39399): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 62ccbd4b99764f078781c3b8d11adff9: Network error: Client connection negotiation failed: client connection to 127.8.158.65:39399: connect: Connection refused (error 111)
I20260929 19:04:32.724150 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 20381
I20260929 19:04:32.729921 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 19995
I20260929 19:04:32.739513 8825 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 19:04:33.085244 8825 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 19:04:33.085348 8825 mini_ranger.cc:66] Stopping Ranger...
I20260929 19:04:33.429454 8825 mini_ranger.cc:68] Stopped Ranger
2026-09-29 19:04:33.429 UTC [19581] LOG: received smart shutdown request
2026-09-29 19:04:33.432 UTC [19581] LOG: background worker "logical replication launcher" (PID 19590) exited with exit code 1
2026-09-29 19:04:33.433 UTC [19585] LOG: shutting down
2026-09-29 19:04:33.604 UTC [19581] LOG: database system is shut down
2026-09-29T19:04:33Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (44915 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:04:33 dist-test-slave-nmsz krb5kdc[20563](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:04:33 dist-test-slave-nmsz krb5kdc[20563](info): set up 2 sockets
Sep 29 19:04:33 dist-test-slave-nmsz krb5kdc[20563](info): commencing operation
krb5kdc: starting...
W20260929 19:04:34.754164 8825 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Sep 29 19:04:34 dist-test-slave-nmsz krb5kdc[20563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:04:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:04:34Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.8.158.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.8.158.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.8.158.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/rangeradmin_127.8.158.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.8.158.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/rangeradmin_127.8.158.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.8.158.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.8.158.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.8.158.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/rangerlookup_127.8.158.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.8.158.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/rangerlookup_127.8.158.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.158.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.8.158.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.158.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/HTTP_127.8.158.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.158.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/HTTP_127.8.158.84@KRBTEST.COM.keytab.
I20260929 19:04:34.905002 8825 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskkdiMiM/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/postgres -l logfile start
2026-09-29 19:04:37.371 UTC [20593] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 19:04:37.371 UTC [20593] LOG: listening on IPv4 address "127.8.158.84", port 45415
2026-09-29 19:04:37.375 UTC [20593] LOG: listening on Unix socket "/tmp/.s.PGSQL.45415"
2026-09-29 19:04:37.409 UTC [20596] LOG: database system was shut down at 2026-09-29 19:04:35 UTC
2026-09-29 19:04:37.413 UTC [20593] LOG: database system is ready to accept connections
I20260929 19:04:38.418210 8825 mini_postgres.cc:96] Postgres bound to 45415
2026-09-29 19:04:38.424 UTC [20604] FATAL: database "slave" does not exist
127.8.158.84:45415 - accepting connections
I20260929 19:04:38.424613 8825 mini_ranger.cc:161] Starting Ranger...
I20260929 19:04:38.438663 8825 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 19:04:38.929772 8825 mini_ranger.cc:87] Created ranger Postgres database
I20260929 19:04:38.929836 8825 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ranger-admin
2026-09-29 19:04:39,200 [I] DB FLAVOR :POSTGRES
2026-09-29 19:04:39,201 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 19:04:39,201 [I] Checking connection..
2026-09-29 19:04:39,201 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.158.84:45415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 19:04:39,436 [I] Checking connection passed.
2026-09-29 19:04:39,436 [I] --------- Verifying version history table ---------
2026-09-29 19:04:39,437 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.158.84:45415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 19:04:39,713 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 19:04:39,714 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.158.84:45415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 19:04:39,962 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 19:04:39,963 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 19:04:39,963 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.158.84:45415/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 19:04:40.195 UTC [20668] WARNING: there is no transaction in progress
2026-09-29 19:04:40,201 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 19:04:40,201 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.158.84:45415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 19:04:40,465 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 19:04:40,465 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 19:04:40,465 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.158.84:45415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 19:04:40,738 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.158.84:45415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 19:04:40,962 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.158.84:45415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-nmsz.c.gcp-upstream.internal','N') ;"
2026-09-29 19:04:41,192 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.158.84:45415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 19:04:41,417 [I] Table x_portal_user does not exist in database ranger
2026-09-29 19:04:41,418 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.158.84:45415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 19:04:41,661 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 19:04:41,661 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.158.84:45415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 19:04:41,881 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.158.84:45415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 19:04:42,109 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 19:04:42,109 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.158.84:45415/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 19:04:43.072 UTC [20776] WARNING: there is no transaction in progress
2026-09-29 19:04:43.089 UTC [20776] WARNING: there is no transaction in progress
2026-09-29 19:04:43.105 UTC [20776] WARNING: there is no transaction in progress
2026-09-29 19:04:43.121 UTC [20776] WARNING: there is no transaction in progress
2026-09-29 19:04:43.137 UTC [20776] WARNING: there is no transaction in progress
2026-09-29 19:04:43.156 UTC [20776] WARNING: there is no transaction in progress
2026-09-29 19:04:43.212 UTC [20776] WARNING: there is no transaction in progress
2026-09-29 19:04:43.219 UTC [20776] WARNING: there is no transaction in progress
2026-09-29 19:04:43.234 UTC [20776] WARNING: there is no transaction in progress
2026-09-29 19:04:43.245 UTC [20776] WARNING: there is no transaction in progress
2026-09-29 19:04:43.256 UTC [20776] WARNING: there is no transaction in progress
2026-09-29 19:04:43.269 UTC [20776] WARNING: there is no transaction in progress
2026-09-29 19:04:43.279 UTC [20776] WARNING: there is no transaction in progress
2026-09-29 19:04:43.286 UTC [20776] WARNING: there is no transaction in progress
2026-09-29 19:04:43.294 UTC [20776] WARNING: there is no transaction in progress
2026-09-29 19:04:43.301 UTC [20776] WARNING: there is no transaction in progress
2026-09-29 19:04:43,981 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 19:04:43,981 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.158.84:45415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 19:04:44,221 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 19:04:44,222 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.158.84:45415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 19:04:44,482 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 19:04:44,483 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.158.84:45415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 19:04:44,738 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 19:04:44,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.158.84:45415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 19:04:44,990 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.158.84:45415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 19:04:45,210 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.158.84:45415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-nmsz.c.gcp-upstream.internal';"
2026-09-29 19:04:45,424 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 19:04:45,425 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.158.84:45415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 19:04:45,670 [I] DB_PATCHES have already been applied
I20260929 19:04:45.675498 8825 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ranger-admin:/tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskkdiMiM/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 19:04:45.675555 8825 mini_ranger.cc:193] Using host: 127.8.158.84
I20260929 19:04:45.678494 8825 mini_ranger.cc:227] Ranger admin URL: http://127.8.158.84:57115
Sep 29, 2026 7:04:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 7:04:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 7:04:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 7:04:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 7:04:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 7:04:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 7:04:46 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 19:04:46.773658 8825 mini_ranger.cc:160] Time spent starting Ranger: real 8.349s user 0.000s sys 0.008s
Sep 29, 2026 7:05:05 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 7:05:05 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 7:05:05 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 7:05:05 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 7:05:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 7:05:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 7:05:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 7:05:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 7:05:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 7:05:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 7:05:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 7:05:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 7:05:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 7:05:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 7:05:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 7:05:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 7:05:05 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 7:05:06 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 19:05:07.230995 8825 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.8.158.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.8.158.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.8.158.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/rangerkms_127.8.158.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.8.158.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/rangerkms_127.8.158.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.158.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.8.158.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.158.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/HTTP_127.8.158.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.158.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/HTTP_127.8.158.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 19:05:07 dist-test-slave-nmsz krb5kdc[20563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708707, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 19:05:07.312893 8825 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 19:05:07.321926 8825 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 19:05:08.235268 8825 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 19:05:08.235353 8825 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ranger-kms
I20260929 19:05:08.235378 8825 mini_ranger_kms.cc:227] Using postgres at 127.8.158.84:45415
2026-09-29 19:05:08,393 [I] DB FLAVOR :POSTGRES
2026-09-29 19:05:08,393 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 19:05:08,393 [I] Checking connection
2026-09-29 19:05:08,393 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.158.84:45415/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 19:05:08,649 [I] connection success
2026-09-29 19:05:08,649 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 19:05:08,649 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 19:05:08,649 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.158.84:45415/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 19:05:08,905 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 19:05:08,905 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 19:05:08,906 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 19:05:08,906 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.158.84:45415/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 19:05:09,162 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 19:05:09.629244 8825 mini_ranger_kms.cc:310] Created kms service
I20260929 19:05:09.818922 8825 mini_ranger_kms.cc:326] Created kudu user
I20260929 19:05:09.868429 8825 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 19:05:09 dist-test-slave-nmsz krb5kdc[20563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708707, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.8.158.84@KRBTEST.COM
I20260929 19:05:10.252314 8825 mini_ranger_kms.cc:384] Added ranger policy
I20260929 19:05:10.252424 8825 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ranger-kms:/tmp/dist-test-taskkdiMiM/build/release/bin/postgresql.jar:/tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskkdiMiM/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 19:05:10.252444 8825 mini_ranger_kms.cc:240] Using host: 127.8.158.84
I20260929 19:05:10.255831 8825 mini_ranger_kms.cc:276] Ranger KMS PID: 20956
I20260929 19:05:10.255906 8825 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.8.158.84:55241
Sep 29, 2026 7:05:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 7:05:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 7:05:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 7:05:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 7:05:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 7:05:11 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 29 19:05:12 dist-test-slave-nmsz krb5kdc[20563](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790708712, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.158.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:05:12.490465 8825 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.178s user 0.000s sys 0.142s
I20260929 19:05:12.490566 8825 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 19:05:12.490610 8825 mini_ranger_kms.cc:401] 127.8.158.84:55241/kms/v1/keys
Sep 29 19:05:12 dist-test-slave-nmsz krb5kdc[20563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708712, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.158.84@KRBTEST.COM for HTTP/127.8.158.84@KRBTEST.COM
Sep 29 19:05:13 dist-test-slave-nmsz krb5kdc[20563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708712, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.158.84@KRBTEST.COM for HTTP/127.8.158.84@KRBTEST.COM
Sep 29 19:05:15 dist-test-slave-nmsz krb5kdc[20563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.8.158.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
I20260929 19:05:15.530843 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:43397
--webserver_interface=127.8.158.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:41889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:43397
--ranger_config_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.158.84:55241/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:05:15.616873 21007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:05:15.617063 21007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:05:15.617084 21007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:05:15.618250 21007 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:05:15.618299 21007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:05:15.618335 21007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:05:15.618373 21007 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:05:15.618403 21007 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:05:15.619868 21007 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:41889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/master-0/wal
--ranger_kms_url=127.8.158.84:55241/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:43397
--ranger_config_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:43397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.158.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:05:15.620090 21007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:05:15.620321 21007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:05:15.625109 21012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:15.625145 21013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:15.625334 21015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:05:15.625234 21007 server_base.cc:1034] running on GCE node
I20260929 19:05:15.625613 21007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:05:15.625976 21007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:05:15.627151 21007 hybrid_clock.cc:648] HybridClock initialized: now 1790708715627127 us; error 41 us; skew 500 ppm
Sep 29 19:05:15 dist-test-slave-nmsz krb5kdc[20563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708715, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:05:15.629940 21007 init.cc:377] Logged in from keytab as kudu/127.8.158.126@KRBTEST.COM (short username kudu)
I20260929 19:05:15.630488 21007 webserver.cc:460] Webserver started at http://127.8.158.126:44577/ using document root <none> and password file <none>
I20260929 19:05:15.630698 21007 fs_manager.cc:359] Metadata directory not provided
I20260929 19:05:15.630769 21007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:05:15.630906 21007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 19:05:15 dist-test-slave-nmsz krb5kdc[20563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708715, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for HTTP/127.8.158.84@KRBTEST.COM
I20260929 19:05:15.687211 21007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "5b0248c724d24afd92a1a9d821b2988e"
format_stamp: "Formatted at 2026-09-29 19:05:15 on dist-test-slave-nmsz"
server_key: "6c2846a4fa1d151dd043723ecf7537eb"
server_key_iv: "8e02a16ebc0075cb1126be8e94f6c3bb"
server_key_version: "kuduclusterkey@0"
I20260929 19:05:15.687616 21007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "5b0248c724d24afd92a1a9d821b2988e"
format_stamp: "Formatted at 2026-09-29 19:05:15 on dist-test-slave-nmsz"
server_key: "6c2846a4fa1d151dd043723ecf7537eb"
server_key_iv: "8e02a16ebc0075cb1126be8e94f6c3bb"
server_key_version: "kuduclusterkey@0"
I20260929 19:05:15.688997 21007 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 19:05:15.704015 21022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:15.704212 21007 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 19:05:15.704320 21007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "5b0248c724d24afd92a1a9d821b2988e"
format_stamp: "Formatted at 2026-09-29 19:05:15 on dist-test-slave-nmsz"
server_key: "6c2846a4fa1d151dd043723ecf7537eb"
server_key_iv: "8e02a16ebc0075cb1126be8e94f6c3bb"
server_key_version: "kuduclusterkey@0"
I20260929 19:05:15.704409 21007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:05:15.724521 21007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:05:15.726516 21007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:05:15.726663 21007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:05:15.731498 21007 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.126:43397
I20260929 19:05:15.731524 21074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.126:43397 every 8 connection(s)
I20260929 19:05:15.731881 21007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:05:15.732611 21075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:05:15.735540 21075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b0248c724d24afd92a1a9d821b2988e: Bootstrap starting.
I20260929 19:05:15.736212 21075 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b0248c724d24afd92a1a9d821b2988e: Neither blocks nor log segments found. Creating new log.
I20260929 19:05:15.736515 21075 log.cc:824] T 00000000000000000000000000000000 P 5b0248c724d24afd92a1a9d821b2988e: Log is configured to *not* fsync() on all Append() calls
I20260929 19:05:15.736981 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 21007
I20260929 19:05:15.737072 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:05:15.737244 8825 external_mini_cluster.cc:1428] Setting key 46026c8ed0373f37fa695814e55f1dc1
I20260929 19:05:15.737260 21075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b0248c724d24afd92a1a9d821b2988e: No bootstrap required, opened a new log
I20260929 19:05:15.740146 21075 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b0248c724d24afd92a1a9d821b2988e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b0248c724d24afd92a1a9d821b2988e" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 43397 } }
I20260929 19:05:15.740325 21075 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b0248c724d24afd92a1a9d821b2988e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:05:15.740360 21075 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b0248c724d24afd92a1a9d821b2988e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b0248c724d24afd92a1a9d821b2988e, State: Initialized, Role: FOLLOWER
I20260929 19:05:15.740504 21075 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b0248c724d24afd92a1a9d821b2988e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b0248c724d24afd92a1a9d821b2988e" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 43397 } }
I20260929 19:05:15.740574 21075 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b0248c724d24afd92a1a9d821b2988e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:05:15.740603 21075 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b0248c724d24afd92a1a9d821b2988e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:05:15.740665 21075 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b0248c724d24afd92a1a9d821b2988e [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:05:15.741299 21075 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b0248c724d24afd92a1a9d821b2988e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b0248c724d24afd92a1a9d821b2988e" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 43397 } }
I20260929 19:05:15.741417 21075 leader_election.cc:302] T 00000000000000000000000000000000 P 5b0248c724d24afd92a1a9d821b2988e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b0248c724d24afd92a1a9d821b2988e; no voters:
I20260929 19:05:15.741590 21075 leader_election.cc:288] T 00000000000000000000000000000000 P 5b0248c724d24afd92a1a9d821b2988e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:05:15.741616 21080 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b0248c724d24afd92a1a9d821b2988e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:05:15.741892 21080 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b0248c724d24afd92a1a9d821b2988e [term 1 LEADER]: Becoming Leader. State: Replica: 5b0248c724d24afd92a1a9d821b2988e, State: Running, Role: LEADER
I20260929 19:05:15.742072 21080 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b0248c724d24afd92a1a9d821b2988e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b0248c724d24afd92a1a9d821b2988e" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 43397 } }
I20260929 19:05:15.741904 21075 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b0248c724d24afd92a1a9d821b2988e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:05:15.743927 21075 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Sep 29 19:05:15 dist-test-slave-nmsz krb5kdc[20563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:05:15.746167 21081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b0248c724d24afd92a1a9d821b2988e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b0248c724d24afd92a1a9d821b2988e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b0248c724d24afd92a1a9d821b2988e" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 43397 } } }
I20260929 19:05:15.746577 21081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b0248c724d24afd92a1a9d821b2988e [sys.catalog]: This master's current role is: LEADER
I20260929 19:05:15.746246 21082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b0248c724d24afd92a1a9d821b2988e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b0248c724d24afd92a1a9d821b2988e. Latest consensus state: current_term: 1 leader_uuid: "5b0248c724d24afd92a1a9d821b2988e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b0248c724d24afd92a1a9d821b2988e" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 43397 } } }
I20260929 19:05:15.747076 21082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b0248c724d24afd92a1a9d821b2988e [sys.catalog]: This master's current role is: LEADER
I20260929 19:05:15.750141 21079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:15.737729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37254 (local address 127.8.158.126:43397)
0929 19:05:15.737890 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:15.737899 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:15.737932 (+ 33us) server_negotiation.cc:407] Connection header received
0929 19:05:15.738818 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:15.738826 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:15.738885 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:15.739029 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:15.739748 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:15.741121 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:15.742190 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:15.742832 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:15.742941 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:15.746197 (+ 3256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:15.746250 (+ 53us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:15.746250 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:15.746382 (+ 132us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:15.748516 (+ 2134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:15.748915 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:15.748917 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:15.748918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:15.748997 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:15.749220 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:15.749221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:15.749221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:15.749558 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:15.749735 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:15.749827 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 19:05:15.749974 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":87,"threads_started":1}
I20260929 19:05:15.754460 21096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:05:15.754974 21096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:05:15.759406 21096 catalog_manager.cc:1347] Generated new cluster ID: 4cde1e4fae2b48a6807eb093e5d2692c
I20260929 19:05:15.759498 21096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:05:15.771592 21096 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:05:15.772334 21096 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:05:15.787421 21096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b0248c724d24afd92a1a9d821b2988e: Generated new TSK 0
I20260929 19:05:15.787655 21096 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.158.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab.
Entry for principal kudu/127.8.158.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab.
I20260929 19:05:16.478639 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.158.84:55241/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.65:0
--local_ip_for_outbound_sockets=127.8.158.65
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:43397
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab
--principal=kudu/127.8.158.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:41889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:05:16.587159 21120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:05:16.587371 21120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:05:16.587407 21120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:05:16.588858 21120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:05:16.588925 21120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:05:16.588970 21120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.65
I20260929 19:05:16.590396 21120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:41889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.8.158.84:55241/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab
--principal=kudu/127.8.158.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:43397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.65
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:05:16.590649 21120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:05:16.590886 21120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:05:16.595806 21128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:16.595889 21125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:16.595814 21126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:05:16.596310 21120 server_base.cc:1034] running on GCE node
I20260929 19:05:16.596459 21120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:05:16.596639 21120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:05:16.597800 21120 hybrid_clock.cc:648] HybridClock initialized: now 1790708716597788 us; error 30 us; skew 500 ppm
Sep 29 19:05:16 dist-test-slave-nmsz krb5kdc[20563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708716, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:05:16.600229 21120 init.cc:377] Logged in from keytab as kudu/127.8.158.65@KRBTEST.COM (short username kudu)
I20260929 19:05:16.600698 21120 webserver.cc:460] Webserver started at http://127.8.158.65:46391/ using document root <none> and password file <none>
I20260929 19:05:16.600932 21120 fs_manager.cc:359] Metadata directory not provided
I20260929 19:05:16.600999 21120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:05:16.601126 21120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 19:05:16 dist-test-slave-nmsz krb5kdc[20563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708716, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for HTTP/127.8.158.84@KRBTEST.COM
I20260929 19:05:16.614648 21120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-0/data/instance:
uuid: "8cf4d02c8b8b44eca96dd6b23d067157"
format_stamp: "Formatted at 2026-09-29 19:05:16 on dist-test-slave-nmsz"
server_key: "648183309fb25d4bdcdae1e9ebefe1a2"
server_key_iv: "899df4032a36c535457bc752631c51f8"
server_key_version: "kuduclusterkey@0"
I20260929 19:05:16.615104 21120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance:
uuid: "8cf4d02c8b8b44eca96dd6b23d067157"
format_stamp: "Formatted at 2026-09-29 19:05:16 on dist-test-slave-nmsz"
server_key: "648183309fb25d4bdcdae1e9ebefe1a2"
server_key_iv: "899df4032a36c535457bc752631c51f8"
server_key_version: "kuduclusterkey@0"
I20260929 19:05:16.616597 21120 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:05:16.628252 21135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:16.628479 21120 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:05:16.628575 21120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-0/wal
uuid: "8cf4d02c8b8b44eca96dd6b23d067157"
format_stamp: "Formatted at 2026-09-29 19:05:16 on dist-test-slave-nmsz"
server_key: "648183309fb25d4bdcdae1e9ebefe1a2"
server_key_iv: "899df4032a36c535457bc752631c51f8"
server_key_version: "kuduclusterkey@0"
I20260929 19:05:16.628676 21120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:05:16.644898 21120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:05:16.648511 21120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:05:16.648746 21120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:05:16.649026 21120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:05:16.649428 21120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:05:16.649511 21120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:16.649574 21120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:05:16.649626 21120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:16.657120 21120 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.65:35713
I20260929 19:05:16.657536 21120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
I20260929 19:05:16.664443 21250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.65:35713 every 8 connection(s)
I20260929 19:05:16.666460 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 21120
I20260929 19:05:16.666570 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance
I20260929 19:05:16.666774 8825 external_mini_cluster.cc:1428] Setting key 4eaba91ab5987761f6f0cbc3c1c5cb88
Sep 29 19:05:16 dist-test-slave-nmsz krb5kdc[20563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708716, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.158.66@KRBTEST.COM; defaulting to no policy
I20260929 19:05:16.679170 21254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:16.665244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:38127 (local address 127.8.158.126:43397)
0929 19:05:16.665417 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:16.665420 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:16.665431 (+ 11us) server_negotiation.cc:407] Connection header received
0929 19:05:16.666154 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:16.666156 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:16.666169 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:16.666258 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:16.669388 (+ 3130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:16.670284 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:16.672126 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:16.672440 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:16.672519 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:16.675333 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:16.675335 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:16.675336 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:16.675344 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:16.677251 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:16.677805 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:16.677806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:16.677807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:16.677876 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:16.678174 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:16.678175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:16.678176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:16.678343 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:16.678394 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:16.678936 (+ 542us) server_negotiation.cc:299] Negotiation successful
0929 19:05:16.679057 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":92,"threads_started":1}
I20260929 19:05:16.679941 21253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:16.664920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:43397 (local address 127.8.158.65:38127)
0929 19:05:16.665117 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:16.665183 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:16.666018 (+ 835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:16.667047 (+ 1029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:16.667051 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:16.667304 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:05:16.669263 (+ 1959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:16.669271 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:16.670528 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:16.670529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:16.671946 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:16.671950 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:16.672559 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:16.672560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:16.672738 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:16.673410 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:05:16.673421 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:05:16.675225 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:05:16.677349 (+ 2124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:16.677356 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:16.677358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:16.677658 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:16.677958 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:16.677959 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:16.677959 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:16.678093 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:16.678446 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:05:16.678452 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:05:16.678615 (+ 163us) client_negotiation.cc:769] Sending connection context
0929 19:05:16.679522 (+ 907us) client_negotiation.cc:240] Negotiation successful
0929 19:05:16.679701 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":89,"threads_started":1}
I20260929 19:05:16.680416 21251 heartbeater.cc:347] Connected to a master server at 127.8.158.126:43397
I20260929 19:05:16.680513 21251 heartbeater.cc:464] Registering TS with master...
I20260929 19:05:16.680706 21251 heartbeater.cc:511] Master 127.8.158.126:43397 requested a full tablet report, sending...
I20260929 19:05:16.681281 21039 ts_manager.cc:194] Registered new tserver with Master: 8cf4d02c8b8b44eca96dd6b23d067157 (127.8.158.65:35713)
I20260929 19:05:16.682178 21039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.65@KRBTEST.COM'} at 127.8.158.65:38127
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab.
Entry for principal kudu/127.8.158.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab.
I20260929 19:05:16.709865 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.158.84:55241/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.66:0
--local_ip_for_outbound_sockets=127.8.158.66
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:43397
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab
--principal=kudu/127.8.158.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:41889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 19:05:16 dist-test-slave-nmsz krb5kdc[20563](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790708716, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 19:05:16.841082 21257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:05:16.841454 21257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:05:16.841540 21257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:05:16.843684 21257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:05:16.843794 21257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:05:16.843883 21257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.66
I20260929 19:05:16.846355 21257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:41889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.8.158.84:55241/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab
--principal=kudu/127.8.158.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:43397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.66
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:05:16.846611 21257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:05:16.846877 21257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:05:16.852993 21265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:05:16.853044 21257 server_base.cc:1034] running on GCE node
W20260929 19:05:16.853010 21267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:16.853404 21264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:05:16.853621 21257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:05:16.853906 21257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:05:16.855064 21257 hybrid_clock.cc:648] HybridClock initialized: now 1790708716855044 us; error 32 us; skew 500 ppm
Sep 29 19:05:16 dist-test-slave-nmsz krb5kdc[20563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708716, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:05:16.858284 21257 init.cc:377] Logged in from keytab as kudu/127.8.158.66@KRBTEST.COM (short username kudu)
I20260929 19:05:16.858763 21257 webserver.cc:460] Webserver started at http://127.8.158.66:39813/ using document root <none> and password file <none>
I20260929 19:05:16.858983 21257 fs_manager.cc:359] Metadata directory not provided
I20260929 19:05:16.859030 21257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:05:16.859141 21257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 19:05:16 dist-test-slave-nmsz krb5kdc[20563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708716, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.66@KRBTEST.COM for HTTP/127.8.158.84@KRBTEST.COM
I20260929 19:05:16.878100 21257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-1/data/instance:
uuid: "7564d80b28f142c898b80127ca0708b6"
format_stamp: "Formatted at 2026-09-29 19:05:16 on dist-test-slave-nmsz"
server_key: "dcb9866c87f360fa98e1935146f06ed9"
server_key_iv: "b6290b0c1137c1060b0588454f6384e6"
server_key_version: "kuduclusterkey@0"
I20260929 19:05:16.878598 21257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance:
uuid: "7564d80b28f142c898b80127ca0708b6"
format_stamp: "Formatted at 2026-09-29 19:05:16 on dist-test-slave-nmsz"
server_key: "dcb9866c87f360fa98e1935146f06ed9"
server_key_iv: "b6290b0c1137c1060b0588454f6384e6"
server_key_version: "kuduclusterkey@0"
I20260929 19:05:16.881062 21257 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 19:05:16.899098 21275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:16.899344 21257 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260929 19:05:16.899430 21257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-1/wal
uuid: "7564d80b28f142c898b80127ca0708b6"
format_stamp: "Formatted at 2026-09-29 19:05:16 on dist-test-slave-nmsz"
server_key: "dcb9866c87f360fa98e1935146f06ed9"
server_key_iv: "b6290b0c1137c1060b0588454f6384e6"
server_key_version: "kuduclusterkey@0"
I20260929 19:05:16.899513 21257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:05:16.934087 21257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:05:16.936549 21257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:05:16.936693 21257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:05:16.936919 21257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:05:16.937299 21257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:05:16.937402 21257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:16.937448 21257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:05:16.937466 21257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:16.944855 21257 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.66:33951
I20260929 19:05:16.945289 21257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
I20260929 19:05:16.946538 21388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.66:33951 every 8 connection(s)
I20260929 19:05:16.947646 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 21257
I20260929 19:05:16.947742 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance
I20260929 19:05:16.947928 8825 external_mini_cluster.cc:1428] Setting key f693ac46add94ad0b2cbb97b6cda44f3
Sep 29 19:05:16 dist-test-slave-nmsz krb5kdc[20563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708716, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.66@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:05:16.956453 21254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:16.945977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:54949 (local address 127.8.158.126:43397)
0929 19:05:16.946025 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:16.946027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:16.946276 (+ 249us) server_negotiation.cc:407] Connection header received
0929 19:05:16.946967 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:16.946968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:16.946983 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:16.947045 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:16.947782 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:16.948553 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:16.950037 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:16.950374 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:16.950423 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:16.952958 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:16.952959 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:16.952960 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:16.952969 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:16.954799 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:16.955318 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:16.955319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:16.955320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:16.955383 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:16.955692 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:16.955693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:16.955693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:16.955845 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:16.955893 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:16.956262 (+ 369us) server_negotiation.cc:299] Negotiation successful
0929 19:05:16.956360 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 19:05:16.957190 21391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:16.945969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:43397 (local address 127.8.158.66:54949)
0929 19:05:16.946125 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:16.946171 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:16.946845 (+ 674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:16.947132 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:16.947137 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:16.947372 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:05:16.947645 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:16.947652 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:16.948678 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:16.948679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:16.949932 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:16.949934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:16.950526 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:16.950527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:16.950660 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:16.951224 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:05:16.951234 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:05:16.952851 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:05:16.954900 (+ 2049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:16.954907 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:16.954909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:16.955218 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:16.955465 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:16.955466 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:16.955466 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:16.955603 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:16.955948 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:05:16.955955 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:05:16.956105 (+ 150us) client_negotiation.cc:769] Sending connection context
0929 19:05:16.956803 (+ 698us) client_negotiation.cc:240] Negotiation successful
0929 19:05:16.956963 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":78,"threads_started":1}
I20260929 19:05:16.957873 21389 heartbeater.cc:347] Connected to a master server at 127.8.158.126:43397
I20260929 19:05:16.958014 21389 heartbeater.cc:464] Registering TS with master...
I20260929 19:05:16.958250 21389 heartbeater.cc:511] Master 127.8.158.126:43397 requested a full tablet report, sending...
I20260929 19:05:16.958674 21039 ts_manager.cc:194] Registered new tserver with Master: 7564d80b28f142c898b80127ca0708b6 (127.8.158.66:33951)
I20260929 19:05:16.959156 21039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.66@KRBTEST.COM'} at 127.8.158.66:54949
WARNING: no policy specified for kudu/127.8.158.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab.
Entry for principal kudu/127.8.158.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab.
I20260929 19:05:16.981614 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.158.84:55241/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.67:0
--local_ip_for_outbound_sockets=127.8.158.67
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:43397
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab
--principal=kudu/127.8.158.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:41889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:05:17.094298 21394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:05:17.094622 21394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:05:17.094662 21394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:05:17.095890 21394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:05:17.095965 21394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:05:17.096009 21394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.67
I20260929 19:05:17.097456 21394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:41889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.8.158.84:55241/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab
--principal=kudu/127.8.158.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:43397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.67
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:05:17.097702 21394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:05:17.097978 21394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:05:17.105530 21402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:17.105482 21400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:05:17.105922 21394 server_base.cc:1034] running on GCE node
W20260929 19:05:17.105546 21399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:05:17.106209 21394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:05:17.106511 21394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:05:17.107661 21394 hybrid_clock.cc:648] HybridClock initialized: now 1790708717107653 us; error 30 us; skew 500 ppm
Sep 29 19:05:17 dist-test-slave-nmsz krb5kdc[20563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708717, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:05:17.110275 21394 init.cc:377] Logged in from keytab as kudu/127.8.158.67@KRBTEST.COM (short username kudu)
I20260929 19:05:17.110792 21394 webserver.cc:460] Webserver started at http://127.8.158.67:40227/ using document root <none> and password file <none>
I20260929 19:05:17.111013 21394 fs_manager.cc:359] Metadata directory not provided
I20260929 19:05:17.111083 21394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:05:17.111217 21394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 19:05:17 dist-test-slave-nmsz krb5kdc[20563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708717, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.67@KRBTEST.COM for HTTP/127.8.158.84@KRBTEST.COM
I20260929 19:05:17.123786 21394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-2/data/instance:
uuid: "234a5283e54b46818e3522c5075446e0"
format_stamp: "Formatted at 2026-09-29 19:05:17 on dist-test-slave-nmsz"
server_key: "856d10a02fbf735c4313753bab8d5e2e"
server_key_iv: "23dde3e02601373dc8999a9997bb43b3"
server_key_version: "kuduclusterkey@0"
I20260929 19:05:17.124224 21394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance:
uuid: "234a5283e54b46818e3522c5075446e0"
format_stamp: "Formatted at 2026-09-29 19:05:17 on dist-test-slave-nmsz"
server_key: "856d10a02fbf735c4313753bab8d5e2e"
server_key_iv: "23dde3e02601373dc8999a9997bb43b3"
server_key_version: "kuduclusterkey@0"
I20260929 19:05:17.137827 21394 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.001s sys 0.002s
I20260929 19:05:17.151638 21409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:17.151870 21394 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 19:05:17.151963 21394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-2/wal
uuid: "234a5283e54b46818e3522c5075446e0"
format_stamp: "Formatted at 2026-09-29 19:05:17 on dist-test-slave-nmsz"
server_key: "856d10a02fbf735c4313753bab8d5e2e"
server_key_iv: "23dde3e02601373dc8999a9997bb43b3"
server_key_version: "kuduclusterkey@0"
I20260929 19:05:17.152060 21394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:05:17.170224 21394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:05:17.172477 21394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:05:17.172623 21394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:05:17.172847 21394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:05:17.173195 21394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:05:17.173256 21394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:17.173295 21394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:05:17.173321 21394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:17.180271 21394 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.67:40495
I20260929 19:05:17.180672 21394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
I20260929 19:05:17.180967 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 21394
I20260929 19:05:17.181079 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance
I20260929 19:05:17.181285 8825 external_mini_cluster.cc:1428] Setting key af473a8a0595597669395f1181a77404
I20260929 19:05:17.182963 21522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.67:40495 every 8 connection(s)
Sep 29 19:05:17 dist-test-slave-nmsz krb5kdc[20563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708717, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.67@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:05:17.193506 21254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:17.181609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:44667 (local address 127.8.158.126:43397)
0929 19:05:17.181652 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:17.181654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:17.182720 (+ 1066us) server_negotiation.cc:407] Connection header received
0929 19:05:17.183577 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:17.183578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:17.183591 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:17.183652 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:17.184324 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:17.185099 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:17.186658 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:17.186954 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:17.187002 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:17.189500 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:17.189501 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:17.189501 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:17.189509 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:17.191265 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:17.191738 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:17.191739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:17.191739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:17.191797 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:17.192088 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:17.192089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:17.192089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:17.192217 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:17.192284 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:17.193319 (+ 1035us) server_negotiation.cc:299] Negotiation successful
0929 19:05:17.193410 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 19:05:17.194024 21525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:17.181394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:43397 (local address 127.8.158.67:44667)
0929 19:05:17.182603 (+ 1209us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:17.182642 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:17.183454 (+ 812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:17.183727 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:17.183731 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:17.183969 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:05:17.184224 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:17.184228 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:17.185233 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:17.185234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:17.186496 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:17.186563 (+ 67us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:17.187106 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:17.187107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:17.187238 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:17.187808 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:05:17.187827 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:05:17.189393 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:05:17.191360 (+ 1967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:17.191366 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:17.191368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:17.191647 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:17.191885 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:17.191885 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:17.191886 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:17.192010 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:17.192995 (+ 985us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:05:17.193001 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:05:17.193154 (+ 153us) client_negotiation.cc:769] Sending connection context
0929 19:05:17.193639 (+ 485us) client_negotiation.cc:240] Negotiation successful
0929 19:05:17.193793 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1172,"thread_start_us":904,"threads_started":1}
I20260929 19:05:17.194468 21523 heartbeater.cc:347] Connected to a master server at 127.8.158.126:43397
I20260929 19:05:17.194566 21523 heartbeater.cc:464] Registering TS with master...
I20260929 19:05:17.194738 21523 heartbeater.cc:511] Master 127.8.158.126:43397 requested a full tablet report, sending...
I20260929 19:05:17.195094 21039 ts_manager.cc:194] Registered new tserver with Master: 234a5283e54b46818e3522c5075446e0 (127.8.158.67:40495)
I20260929 19:05:17.195560 21039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.67@KRBTEST.COM'} at 127.8.158.67:44667
I20260929 19:05:17.203761 8825 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:05:17.204591 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 21120
I20260929 19:05:17.211030 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 21257
I20260929 19:05:17.216392 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 21394
I20260929 19:05:17.222450 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 21007
I20260929 19:05:17.230769 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:43397
--webserver_interface=127.8.158.126
--webserver_port=44577
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:41889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:43397
--ranger_config_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.158.84:55241/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.158.84:55241/kms with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:05:17.309640 21526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:05:17.309834 21526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:05:17.309857 21526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:05:17.311056 21526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:05:17.311097 21526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:05:17.311112 21526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:05:17.311151 21526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:05:17.311182 21526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:05:17.312660 21526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:41889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.158.84:55241/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:43397
--ranger_config_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:43397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.158.126
--webserver_port=44577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:05:17.312903 21526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:05:17.313112 21526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:05:17.317781 21531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:17.317813 21534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:17.317866 21532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:05:17.317893 21526 server_base.cc:1034] running on GCE node
I20260929 19:05:17.318189 21526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:05:17.318509 21526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:05:17.319653 21526 hybrid_clock.cc:648] HybridClock initialized: now 1790708717319631 us; error 37 us; skew 500 ppm
Sep 29 19:05:17 dist-test-slave-nmsz krb5kdc[20563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708717, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:05:17.321986 21526 init.cc:377] Logged in from keytab as kudu/127.8.158.126@KRBTEST.COM (short username kudu)
I20260929 19:05:17.322499 21526 webserver.cc:460] Webserver started at http://127.8.158.126:44577/ using document root <none> and password file <none>
I20260929 19:05:17.322726 21526 fs_manager.cc:359] Metadata directory not provided
I20260929 19:05:17.322804 21526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 19:05:17 dist-test-slave-nmsz krb5kdc[20563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708717, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for HTTP/127.8.158.84@KRBTEST.COM
I20260929 19:05:17.386646 21526 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:05:17.387380 21542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:17.387562 21526 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:05:17.387645 21526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "5b0248c724d24afd92a1a9d821b2988e"
format_stamp: "Formatted at 2026-09-29 19:05:15 on dist-test-slave-nmsz"
server_key: "6c2846a4fa1d151dd043723ecf7537eb"
server_key_iv: "8e02a16ebc0075cb1126be8e94f6c3bb"
server_key_version: "kuduclusterkey@0"
I20260929 19:05:17.388027 21526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:05:17.403426 21526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:05:17.405215 21526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:05:17.405378 21526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:05:17.409538 21526 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.126:43397
I20260929 19:05:17.409592 21594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.126:43397 every 8 connection(s)
I20260929 19:05:17.409982 21526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:05:17.411144 21595 sys_catalog.cc:263] Verifying existing consensus state
I20260929 19:05:17.411680 21595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b0248c724d24afd92a1a9d821b2988e: Bootstrap starting.
I20260929 19:05:17.416465 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 21526
I20260929 19:05:17.416787 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.158.84:55241/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.65:35713
--local_ip_for_outbound_sockets=127.8.158.65
--tserver_master_addrs=127.8.158.126:43397
--webserver_port=46391
--webserver_interface=127.8.158.65
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab
--principal=kudu/127.8.158.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:41889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.158.84:55241/kms with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:05:17.419746 21595 log.cc:824] T 00000000000000000000000000000000 P 5b0248c724d24afd92a1a9d821b2988e: Log is configured to *not* fsync() on all Append() calls
I20260929 19:05:17.421309 21595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b0248c724d24afd92a1a9d821b2988e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 19:05:17.421594 21595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b0248c724d24afd92a1a9d821b2988e: Bootstrap complete.
I20260929 19:05:17.423988 21595 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b0248c724d24afd92a1a9d821b2988e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b0248c724d24afd92a1a9d821b2988e" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 43397 } }
I20260929 19:05:17.424208 21595 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b0248c724d24afd92a1a9d821b2988e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b0248c724d24afd92a1a9d821b2988e, State: Initialized, Role: FOLLOWER
I20260929 19:05:17.424360 21595 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b0248c724d24afd92a1a9d821b2988e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b0248c724d24afd92a1a9d821b2988e" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 43397 } }
I20260929 19:05:17.424428 21595 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b0248c724d24afd92a1a9d821b2988e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:05:17.424499 21595 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b0248c724d24afd92a1a9d821b2988e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:05:17.424543 21595 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b0248c724d24afd92a1a9d821b2988e [term 1 FOLLOWER]: Advancing to term 2
I20260929 19:05:17.425540 21595 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b0248c724d24afd92a1a9d821b2988e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b0248c724d24afd92a1a9d821b2988e" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 43397 } }
I20260929 19:05:17.425625 21595 leader_election.cc:302] T 00000000000000000000000000000000 P 5b0248c724d24afd92a1a9d821b2988e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b0248c724d24afd92a1a9d821b2988e; no voters:
I20260929 19:05:17.425776 21595 leader_election.cc:288] T 00000000000000000000000000000000 P 5b0248c724d24afd92a1a9d821b2988e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 19:05:17.425813 21599 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b0248c724d24afd92a1a9d821b2988e [term 2 FOLLOWER]: Leader election won for term 2
I20260929 19:05:17.425915 21599 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b0248c724d24afd92a1a9d821b2988e [term 2 LEADER]: Becoming Leader. State: Replica: 5b0248c724d24afd92a1a9d821b2988e, State: Running, Role: LEADER
I20260929 19:05:17.426035 21595 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b0248c724d24afd92a1a9d821b2988e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:05:17.426206 21599 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b0248c724d24afd92a1a9d821b2988e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b0248c724d24afd92a1a9d821b2988e" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 43397 } }
I20260929 19:05:17.426606 21600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b0248c724d24afd92a1a9d821b2988e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5b0248c724d24afd92a1a9d821b2988e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b0248c724d24afd92a1a9d821b2988e" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 43397 } } }
I20260929 19:05:17.426692 21600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b0248c724d24afd92a1a9d821b2988e [sys.catalog]: This master's current role is: LEADER
I20260929 19:05:17.426851 21601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b0248c724d24afd92a1a9d821b2988e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b0248c724d24afd92a1a9d821b2988e. Latest consensus state: current_term: 2 leader_uuid: "5b0248c724d24afd92a1a9d821b2988e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b0248c724d24afd92a1a9d821b2988e" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 43397 } } }
I20260929 19:05:17.426908 21601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b0248c724d24afd92a1a9d821b2988e [sys.catalog]: This master's current role is: LEADER
I20260929 19:05:17.426990 21608 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:05:17.427546 21595 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 19:05:17.427616 21608 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:05:17.427737 21608 catalog_manager.cc:1259] Loaded cluster ID: 4cde1e4fae2b48a6807eb093e5d2692c
I20260929 19:05:17.427773 21608 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:05:17.431421 21608 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:05:17.431906 21608 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5b0248c724d24afd92a1a9d821b2988e: Loaded TSK: 0
I20260929 19:05:17.432154 21608 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 19:05:17.503854 21597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:05:17.504066 21597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:05:17.504107 21597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:05:17.505254 21597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:05:17.505316 21597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:05:17.505360 21597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.65
I20260929 19:05:17.507120 21597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:41889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.158.84:55241/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab
--principal=kudu/127.8.158.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.65:35713
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.158.65
--webserver_port=46391
--tserver_master_addrs=127.8.158.126:43397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.65
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:05:17.507390 21597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:05:17.507617 21597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:05:17.512604 21632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:17.512637 21635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:17.512928 21633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:05:17.513846 21597 server_base.cc:1034] running on GCE node
I20260929 19:05:17.514055 21597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:05:17.514323 21597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:05:17.515507 21597 hybrid_clock.cc:648] HybridClock initialized: now 1790708717515491 us; error 24 us; skew 500 ppm
Sep 29 19:05:17 dist-test-slave-nmsz krb5kdc[20563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708717, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:05:17.517992 21597 init.cc:377] Logged in from keytab as kudu/127.8.158.65@KRBTEST.COM (short username kudu)
I20260929 19:05:17.518486 21597 webserver.cc:460] Webserver started at http://127.8.158.65:46391/ using document root <none> and password file <none>
I20260929 19:05:17.518725 21597 fs_manager.cc:359] Metadata directory not provided
I20260929 19:05:17.518791 21597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 19:05:17 dist-test-slave-nmsz krb5kdc[20563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708717, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for HTTP/127.8.158.84@KRBTEST.COM
I20260929 19:05:17.550707 21597 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.000s
I20260929 19:05:17.553792 21643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:17.553995 21597 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.000s
I20260929 19:05:17.554076 21597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-0/wal
uuid: "8cf4d02c8b8b44eca96dd6b23d067157"
format_stamp: "Formatted at 2026-09-29 19:05:16 on dist-test-slave-nmsz"
server_key: "648183309fb25d4bdcdae1e9ebefe1a2"
server_key_iv: "899df4032a36c535457bc752631c51f8"
server_key_version: "kuduclusterkey@0"
I20260929 19:05:17.554557 21597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:05:17.565354 21597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:05:17.567648 21597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:05:17.567816 21597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:05:17.568030 21597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:05:17.568390 21597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:05:17.568434 21597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:17.568468 21597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:05:17.568487 21597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:17.575376 21597 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.65:35713
I20260929 19:05:17.575839 21597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
I20260929 19:05:17.576764 21756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.65:35713 every 8 connection(s)
I20260929 19:05:17.582084 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 21597
I20260929 19:05:17.582410 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.158.84:55241/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.66:33951
--local_ip_for_outbound_sockets=127.8.158.66
--tserver_master_addrs=127.8.158.126:43397
--webserver_port=39813
--webserver_interface=127.8.158.66
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab
--principal=kudu/127.8.158.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:41889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.158.84:55241/kms with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 19:05:17 dist-test-slave-nmsz krb5kdc[20563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708717, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:05:17.602977 21760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:17.577016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:52421 (local address 127.8.158.126:43397)
0929 19:05:17.577261 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:17.577269 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:17.583244 (+ 5975us) server_negotiation.cc:407] Connection header received
0929 19:05:17.583387 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:17.583393 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:17.583446 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:17.583573 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:17.584419 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:17.585880 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:17.591742 (+ 5862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:17.592230 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:17.592365 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:17.598961 (+ 6596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:17.598965 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:17.598965 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:17.598977 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:17.601053 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:17.601540 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:17.601544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:17.601544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:17.601641 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:17.601895 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:17.601896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:17.601896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:17.602157 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:17.602220 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:17.602622 (+ 402us) server_negotiation.cc:299] Negotiation successful
0929 19:05:17.602750 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":97,"threads_started":1}
I20260929 19:05:17.603745 21759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:17.576617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:43397 (local address 127.8.158.65:52421)
0929 19:05:17.582394 (+ 5777us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:17.582443 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:17.583082 (+ 639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:17.583757 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:17.583761 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:17.584014 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:05:17.584282 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:17.584289 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:17.590378 (+ 6089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:17.590379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:17.591621 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:17.591625 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:17.592352 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:17.592354 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:17.592471 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:17.593076 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:05:17.593094 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:05:17.598852 (+ 5758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:05:17.601144 (+ 2292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:17.601150 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:17.601152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:17.601438 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:17.601710 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:17.601711 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:17.601711 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:17.601823 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:17.602265 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:05:17.602270 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:05:17.602441 (+ 171us) client_negotiation.cc:769] Sending connection context
0929 19:05:17.603345 (+ 904us) client_negotiation.cc:240] Negotiation successful
0929 19:05:17.603493 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5733,"thread_start_us":85,"threads_started":1}
I20260929 19:05:17.604612 21757 heartbeater.cc:347] Connected to a master server at 127.8.158.126:43397
I20260929 19:05:17.604693 21757 heartbeater.cc:464] Registering TS with master...
I20260929 19:05:17.604858 21757 heartbeater.cc:511] Master 127.8.158.126:43397 requested a full tablet report, sending...
W20260929 19:05:17.605669 21757 heartbeater.cc:644] Failed to heartbeat to 127.8.158.126:43397 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 19:05:17.605727 21757 heartbeater.cc:464] Registering TS with master...
I20260929 19:05:17.605783 21757 heartbeater.cc:511] Master 127.8.158.126:43397 requested a full tablet report, sending...
I20260929 19:05:17.606139 21757 heartbeater.cc:464] Registering TS with master...
I20260929 19:05:17.606204 21757 heartbeater.cc:511] Master 127.8.158.126:43397 requested a full tablet report, sending...
W20260929 19:05:17.606562 21757 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 19:05:17.689322 21761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:05:17.689594 21761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:05:17.689662 21761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:05:17.691123 21761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:05:17.691182 21761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:05:17.691226 21761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.66
I20260929 19:05:17.692687 21761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:41889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.158.84:55241/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab
--principal=kudu/127.8.158.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.66:33951
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.158.66
--webserver_port=39813
--tserver_master_addrs=127.8.158.126:43397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.66
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:05:17.692929 21761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:05:17.693176 21761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:05:17.699272 21766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:05:17.699422 21761 server_base.cc:1034] running on GCE node
W20260929 19:05:17.700222 21769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:17.700419 21767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:05:17.700632 21761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:05:17.700897 21761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:05:17.702191 21761 hybrid_clock.cc:648] HybridClock initialized: now 1790708717702179 us; error 28 us; skew 500 ppm
Sep 29 19:05:17 dist-test-slave-nmsz krb5kdc[20563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708717, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:05:17.705093 21761 init.cc:377] Logged in from keytab as kudu/127.8.158.66@KRBTEST.COM (short username kudu)
I20260929 19:05:17.705629 21761 webserver.cc:460] Webserver started at http://127.8.158.66:39813/ using document root <none> and password file <none>
I20260929 19:05:17.705943 21761 fs_manager.cc:359] Metadata directory not provided
I20260929 19:05:17.706064 21761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 19:05:17 dist-test-slave-nmsz krb5kdc[20563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708717, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.66@KRBTEST.COM for HTTP/127.8.158.84@KRBTEST.COM
I20260929 19:05:17.729746 21761 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.001s
I20260929 19:05:17.730672 21777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:17.730839 21761 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:05:17.730937 21761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-1/wal
uuid: "7564d80b28f142c898b80127ca0708b6"
format_stamp: "Formatted at 2026-09-29 19:05:16 on dist-test-slave-nmsz"
server_key: "dcb9866c87f360fa98e1935146f06ed9"
server_key_iv: "b6290b0c1137c1060b0588454f6384e6"
server_key_version: "kuduclusterkey@0"
I20260929 19:05:17.731299 21761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:05:17.755863 21761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:05:17.757995 21761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:05:17.758154 21761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:05:17.758430 21761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:05:17.758771 21761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:05:17.758836 21761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:17.758903 21761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:05:17.758931 21761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:17.766029 21761 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.66:33951
I20260929 19:05:17.766120 21890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.66:33951 every 8 connection(s)
I20260929 19:05:17.766484 21761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
I20260929 19:05:17.768532 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 21761
I20260929 19:05:17.768810 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.158.84:55241/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.67:40495
--local_ip_for_outbound_sockets=127.8.158.67
--tserver_master_addrs=127.8.158.126:43397
--webserver_port=40227
--webserver_interface=127.8.158.67
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab
--principal=kudu/127.8.158.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:41889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.158.84:55241/kms with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 19:05:17 dist-test-slave-nmsz krb5kdc[20563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708717, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.66@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:05:17.777029 21760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:17.767245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:45977 (local address 127.8.158.126:43397)
0929 19:05:17.767288 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:17.767290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:17.767557 (+ 267us) server_negotiation.cc:407] Connection header received
0929 19:05:17.768220 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:17.768222 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:17.768238 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:17.768310 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:17.768972 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:17.769746 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:17.771254 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:17.771545 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:17.771593 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:17.774047 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:17.774048 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:17.774049 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:17.774057 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:17.775440 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:17.776005 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:17.776007 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:17.776007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:17.776061 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:17.776326 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:17.776327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:17.776327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:17.776472 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:17.776534 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:17.776850 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 19:05:17.776940 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 19:05:17.777371 21893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:17.767252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:43397 (local address 127.8.158.66:45977)
0929 19:05:17.767429 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:17.767475 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:17.768093 (+ 618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:17.768374 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:17.768378 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:17.768594 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:05:17.768857 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:17.768863 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:17.769861 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:17.769862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:17.771135 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:17.771138 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:17.771700 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:17.771701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:17.771842 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:17.772422 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:05:17.772431 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:05:17.773949 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:05:17.775701 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:17.775706 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:17.775707 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:17.775912 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:17.776133 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:17.776134 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:17.776134 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:17.776247 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:17.776562 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:05:17.776568 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:05:17.776708 (+ 140us) client_negotiation.cc:769] Sending connection context
0929 19:05:17.776818 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 19:05:17.776955 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":95,"threads_started":1}
I20260929 19:05:17.778008 21891 heartbeater.cc:347] Connected to a master server at 127.8.158.126:43397
I20260929 19:05:17.778101 21891 heartbeater.cc:464] Registering TS with master...
I20260929 19:05:17.778292 21891 heartbeater.cc:511] Master 127.8.158.126:43397 requested a full tablet report, sending...
W20260929 19:05:17.779074 21891 heartbeater.cc:644] Failed to heartbeat to 127.8.158.126:43397 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 19:05:17.779148 21891 heartbeater.cc:464] Registering TS with master...
I20260929 19:05:17.779215 21891 heartbeater.cc:511] Master 127.8.158.126:43397 requested a full tablet report, sending...
I20260929 19:05:17.779525 21891 heartbeater.cc:464] Registering TS with master...
I20260929 19:05:17.779603 21891 heartbeater.cc:511] Master 127.8.158.126:43397 requested a full tablet report, sending...
W20260929 19:05:17.779886 21891 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 19:05:17.884812 21894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:05:17.885068 21894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:05:17.885107 21894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:05:17.886788 21894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:05:17.886840 21894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:05:17.886878 21894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.67
I20260929 19:05:17.889096 21894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:41889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.158.84:55241/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab
--principal=kudu/127.8.158.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.67:40495
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.158.67
--webserver_port=40227
--tserver_master_addrs=127.8.158.126:43397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.67
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:05:17.889320 21894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:05:17.889542 21894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:05:17.895643 21900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:17.895660 21899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:17.896275 21902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:05:17.896466 21894 server_base.cc:1034] running on GCE node
I20260929 19:05:17.896638 21894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:05:17.896903 21894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:05:17.898084 21894 hybrid_clock.cc:648] HybridClock initialized: now 1790708717898049 us; error 50 us; skew 500 ppm
Sep 29 19:05:17 dist-test-slave-nmsz krb5kdc[20563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708717, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:05:17.901216 21894 init.cc:377] Logged in from keytab as kudu/127.8.158.67@KRBTEST.COM (short username kudu)
I20260929 19:05:17.901700 21894 webserver.cc:460] Webserver started at http://127.8.158.67:40227/ using document root <none> and password file <none>
I20260929 19:05:17.901961 21894 fs_manager.cc:359] Metadata directory not provided
I20260929 19:05:17.902045 21894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 19:05:17 dist-test-slave-nmsz krb5kdc[20563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708717, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.67@KRBTEST.COM for HTTP/127.8.158.84@KRBTEST.COM
I20260929 19:05:17.926743 21894 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260929 19:05:17.930769 21910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:17.930976 21894 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.000s
I20260929 19:05:17.931053 21894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-2/wal
uuid: "234a5283e54b46818e3522c5075446e0"
format_stamp: "Formatted at 2026-09-29 19:05:17 on dist-test-slave-nmsz"
server_key: "856d10a02fbf735c4313753bab8d5e2e"
server_key_iv: "23dde3e02601373dc8999a9997bb43b3"
server_key_version: "kuduclusterkey@0"
I20260929 19:05:17.931488 21894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:05:17.949046 21894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:05:17.951030 21894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:05:17.951170 21894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:05:17.951409 21894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:05:17.951776 21894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:05:17.951817 21894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:17.951839 21894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:05:17.951853 21894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:17.959301 21894 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.67:40495
I20260929 19:05:17.959630 22024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.67:40495 every 8 connection(s)
I20260929 19:05:17.959676 21894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
Sep 29 19:05:17 dist-test-slave-nmsz krb5kdc[20563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708717, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.67@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:05:17.966702 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 21894
I20260929 19:05:17.971096 22027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:17.960406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:43397 (local address 127.8.158.67:37817)
0929 19:05:17.960561 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:17.960601 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:17.961182 (+ 581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:17.961481 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:17.961484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:17.961676 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:05:17.961932 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:17.961937 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:17.962927 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:17.962928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:17.963673 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:17.963675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:17.964186 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:17.964187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:17.964331 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:17.964876 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:05:17.964885 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:05:17.966561 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:05:17.968434 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:17.968441 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:17.968443 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:17.968745 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:17.969942 (+ 1197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:17.969943 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:17.969943 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:17.970078 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:17.970434 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:05:17.970441 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:05:17.970599 (+ 158us) client_negotiation.cc:769] Sending connection context
0929 19:05:17.970716 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 19:05:17.970871 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
I20260929 19:05:17.971426 21760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:17.960644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:37817 (local address 127.8.158.126:43397)
0929 19:05:17.960737 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:17.960740 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:17.960754 (+ 14us) server_negotiation.cc:407] Connection header received
0929 19:05:17.961328 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:17.961330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:17.961343 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:17.961430 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:17.962042 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:17.962841 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:17.963771 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:17.964062 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:17.964145 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:17.966679 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:17.966681 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:17.966681 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:17.966688 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:17.968337 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:17.968843 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:17.968845 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:17.968845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:17.968908 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:17.970166 (+ 1258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:17.970167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:17.970168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:17.970335 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:17.970404 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:17.971264 (+ 860us) server_negotiation.cc:299] Negotiation successful
0929 19:05:17.971342 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 19:05:17.972076 22025 heartbeater.cc:347] Connected to a master server at 127.8.158.126:43397
I20260929 19:05:17.972180 22025 heartbeater.cc:464] Registering TS with master...
I20260929 19:05:17.972373 22025 heartbeater.cc:511] Master 127.8.158.126:43397 requested a full tablet report, sending...
W20260929 19:05:17.973343 22025 heartbeater.cc:644] Failed to heartbeat to 127.8.158.126:43397 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 19:05:17.973423 22025 heartbeater.cc:464] Registering TS with master...
I20260929 19:05:17.973491 22025 heartbeater.cc:511] Master 127.8.158.126:43397 requested a full tablet report, sending...
I20260929 19:05:17.974148 22025 heartbeater.cc:464] Registering TS with master...
I20260929 19:05:17.974231 22025 heartbeater.cc:511] Master 127.8.158.126:43397 requested a full tablet report, sending...
W20260929 19:05:17.974647 22025 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 19:05:17.975792 22028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:17.967058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37270 (local address 127.8.158.126:43397)
0929 19:05:17.967485 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:17.967487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:17.968719 (+ 1232us) server_negotiation.cc:407] Connection header received
0929 19:05:17.968811 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:17.968812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:17.968824 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:17.968910 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:17.969486 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:17.969967 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:17.970813 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:17.971017 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:17.971103 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:17.972834 (+ 1731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:17.972835 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:17.972835 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:17.972842 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:17.974620 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:17.974990 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:17.974991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:17.974992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:17.975047 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:17.975266 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:17.975266 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:17.975267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:17.975372 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:17.975550 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:17.975585 (+ 35us) server_negotiation.cc:299] Negotiation successful
0929 19:05:17.975659 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":54,"threads_started":1}
Sep 29 19:05:18 dist-test-slave-nmsz krb5kdc[20563](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790708718, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:05:18.606729 21757 heartbeater.cc:464] Registering TS with master...
I20260929 19:05:18.606879 21757 heartbeater.cc:511] Master 127.8.158.126:43397 requested a full tablet report, sending...
I20260929 19:05:18.607338 21559 ts_manager.cc:194] Registered new tserver with Master: 8cf4d02c8b8b44eca96dd6b23d067157 (127.8.158.65:35713)
I20260929 19:05:18.608361 21559 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.65@KRBTEST.COM'} at 127.8.158.65:52421
I20260929 19:05:18.780078 21891 heartbeater.cc:464] Registering TS with master...
I20260929 19:05:18.780224 21891 heartbeater.cc:511] Master 127.8.158.126:43397 requested a full tablet report, sending...
I20260929 19:05:18.780493 21559 ts_manager.cc:194] Registered new tserver with Master: 7564d80b28f142c898b80127ca0708b6 (127.8.158.66:33951)
I20260929 19:05:18.781204 21559 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.66@KRBTEST.COM'} at 127.8.158.66:45977
Sep 29 19:05:18 dist-test-slave-nmsz krb5kdc[20563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708718, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for HTTP/127.8.158.84@KRBTEST.COM
I20260929 19:05:18.974845 22025 heartbeater.cc:464] Registering TS with master...
I20260929 19:05:18.975003 22025 heartbeater.cc:511] Master 127.8.158.126:43397 requested a full tablet report, sending...
I20260929 19:05:18.975358 21559 ts_manager.cc:194] Registered new tserver with Master: 234a5283e54b46818e3522c5075446e0 (127.8.158.67:40495)
I20260929 19:05:18.975870 21559 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.67@KRBTEST.COM'} at 127.8.158.67:37817
I20260929 19:05:18.979929 8825 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:05:18.988408 22047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:18.980895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37272 (local address 127.8.158.126:43397)
0929 19:05:18.981006 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:18.981008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:18.981021 (+ 13us) server_negotiation.cc:407] Connection header received
0929 19:05:18.981071 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:18.981072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:18.981084 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:18.981169 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:18.981795 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:18.982590 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:18.983567 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:18.983888 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:18.983963 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:18.985439 (+ 1476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:18.985441 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:18.985441 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:18.985450 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:18.987198 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:18.987581 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:18.987582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:18.987582 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:18.987651 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:18.987854 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:18.987855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:18.987856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:18.987997 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:18.988106 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:18.988213 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 19:05:18.988300 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":61,"threads_started":1}
I20260929 19:05:18.990792 21559 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:05:18.993188 21559 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:05:19.002769 22055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:18.996433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59466 (local address 127.8.158.65:35713)
0929 19:05:18.996582 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:18.996587 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:18.998542 (+ 1955us) server_negotiation.cc:407] Connection header received
0929 19:05:18.998689 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:18.998691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:18.998707 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:18.998755 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:05:18.999000 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:19.000046 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:19.001425 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:19.002299 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:19.002360 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:19.002415 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:19.002654 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 19:05:19.002694 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":60,"threads_started":1}
I20260929 19:05:19.003187 22052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:18.996163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:35713 (local address 127.0.0.1:59466)
0929 19:05:18.998432 (+ 2269us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:18.998471 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:18.998589 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:18.998864 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:18.998866 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:18.998868 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:05:18.998926 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:18.998928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:19.000143 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:19.000144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:19.001324 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:19.001326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:19.002471 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:19.002471 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:19.002545 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:19.002557 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 19:05:19.003053 (+ 496us) client_negotiation.cc:240] Negotiation successful
0929 19:05:19.003115 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2239,"thread_start_us":91,"threads_started":1}
I20260929 19:05:19.004082 21673 tablet_service.cc:1467] Processing CreateTablet for tablet ca1789938dba43fba9d4e3075bc5a799 (DEFAULT_TABLE table=test-table [id=5fddcccc7ddb4ef099ac8c537e9df41c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:05:19.004441 21673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca1789938dba43fba9d4e3075bc5a799. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:05:19.007125 22054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:18.996384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.67:40495 (local address 127.0.0.1:50204)
0929 19:05:18.998987 (+ 2603us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:18.998998 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:18.999040 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:18.999657 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:18.999658 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:18.999659 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:05:18.999684 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:18.999685 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:19.003539 (+ 3854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:19.003540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:19.004714 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:19.004717 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:19.006917 (+ 2200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:19.006918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:19.006977 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:19.006980 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:05:19.007022 (+ 42us) client_negotiation.cc:240] Negotiation successful
0929 19:05:19.007056 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2586,"thread_start_us":59,"threads_started":1}
I20260929 19:05:19.007333 22056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:18.996487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50204 (local address 127.8.158.67:40495)
0929 19:05:18.998444 (+ 1957us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:18.998448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:18.999104 (+ 656us) server_negotiation.cc:407] Connection header received
0929 19:05:18.999142 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:18.999144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:18.999160 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:18.999230 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:05:18.999754 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:19.000781 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:19.004815 (+ 4034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:19.005581 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:19.005653 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:19.005705 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:19.007221 (+ 1516us) server_negotiation.cc:299] Negotiation successful
0929 19:05:19.007262 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1922,"thread_start_us":44,"threads_started":1}
I20260929 19:05:19.007601 22057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:18.996562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32964 (local address 127.8.158.66:33951)
0929 19:05:18.996690 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:18.996693 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:18.998857 (+ 2164us) server_negotiation.cc:407] Connection header received
0929 19:05:18.999367 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:18.999370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:18.999389 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:18.999441 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:05:19.001190 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:19.002233 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:19.005769 (+ 3536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:19.006545 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:19.006650 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:19.006706 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:19.007479 (+ 773us) server_negotiation.cc:299] Negotiation successful
0929 19:05:19.007522 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":54,"threads_started":1}
I20260929 19:05:19.007807 22053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:18.996167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.66:33951 (local address 127.0.0.1:32964)
0929 19:05:18.998505 (+ 2338us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:18.998795 (+ 290us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:18.998901 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:18.999558 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:18.999559 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:18.999561 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:05:18.999599 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:18.999600 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:19.002358 (+ 2758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:19.002358 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:19.003454 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:19.003457 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:19.006770 (+ 3313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:19.006771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:19.006832 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:19.006835 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:05:19.007704 (+ 869us) client_negotiation.cc:240] Negotiation successful
0929 19:05:19.007745 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2318,"thread_start_us":86,"threads_started":1}
I20260929 19:05:19.007933 21959 tablet_service.cc:1467] Processing CreateTablet for tablet ca1789938dba43fba9d4e3075bc5a799 (DEFAULT_TABLE table=test-table [id=5fddcccc7ddb4ef099ac8c537e9df41c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:05:19.008248 21959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca1789938dba43fba9d4e3075bc5a799. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:05:19.008455 21825 tablet_service.cc:1467] Processing CreateTablet for tablet ca1789938dba43fba9d4e3075bc5a799 (DEFAULT_TABLE table=test-table [id=5fddcccc7ddb4ef099ac8c537e9df41c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:05:19.008534 22058 tablet_bootstrap.cc:492] T ca1789938dba43fba9d4e3075bc5a799 P 8cf4d02c8b8b44eca96dd6b23d067157: Bootstrap starting.
I20260929 19:05:19.008773 21825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca1789938dba43fba9d4e3075bc5a799. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:05:19.009227 22058 tablet_bootstrap.cc:654] T ca1789938dba43fba9d4e3075bc5a799 P 8cf4d02c8b8b44eca96dd6b23d067157: Neither blocks nor log segments found. Creating new log.
I20260929 19:05:19.009539 22058 log.cc:824] T ca1789938dba43fba9d4e3075bc5a799 P 8cf4d02c8b8b44eca96dd6b23d067157: Log is configured to *not* fsync() on all Append() calls
I20260929 19:05:19.010390 22058 tablet_bootstrap.cc:492] T ca1789938dba43fba9d4e3075bc5a799 P 8cf4d02c8b8b44eca96dd6b23d067157: No bootstrap required, opened a new log
I20260929 19:05:19.010465 22058 ts_tablet_manager.cc:1397] T ca1789938dba43fba9d4e3075bc5a799 P 8cf4d02c8b8b44eca96dd6b23d067157: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:05:19.012046 22060 tablet_bootstrap.cc:492] T ca1789938dba43fba9d4e3075bc5a799 P 234a5283e54b46818e3522c5075446e0: Bootstrap starting.
I20260929 19:05:19.012599 22060 tablet_bootstrap.cc:654] T ca1789938dba43fba9d4e3075bc5a799 P 234a5283e54b46818e3522c5075446e0: Neither blocks nor log segments found. Creating new log.
I20260929 19:05:19.012882 22060 log.cc:824] T ca1789938dba43fba9d4e3075bc5a799 P 234a5283e54b46818e3522c5075446e0: Log is configured to *not* fsync() on all Append() calls
I20260929 19:05:19.013012 22058 raft_consensus.cc:348] T ca1789938dba43fba9d4e3075bc5a799 P 8cf4d02c8b8b44eca96dd6b23d067157 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf4d02c8b8b44eca96dd6b23d067157" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 35713 } } peers { permanent_uuid: "7564d80b28f142c898b80127ca0708b6" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 33951 } } peers { permanent_uuid: "234a5283e54b46818e3522c5075446e0" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 40495 } }
I20260929 19:05:19.013159 22061 tablet_bootstrap.cc:492] T ca1789938dba43fba9d4e3075bc5a799 P 7564d80b28f142c898b80127ca0708b6: Bootstrap starting.
I20260929 19:05:19.013181 22058 raft_consensus.cc:374] T ca1789938dba43fba9d4e3075bc5a799 P 8cf4d02c8b8b44eca96dd6b23d067157 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:05:19.013218 22058 raft_consensus.cc:729] T ca1789938dba43fba9d4e3075bc5a799 P 8cf4d02c8b8b44eca96dd6b23d067157 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cf4d02c8b8b44eca96dd6b23d067157, State: Initialized, Role: FOLLOWER
I20260929 19:05:19.013348 22058 consensus_queue.cc:260] T ca1789938dba43fba9d4e3075bc5a799 P 8cf4d02c8b8b44eca96dd6b23d067157 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf4d02c8b8b44eca96dd6b23d067157" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 35713 } } peers { permanent_uuid: "7564d80b28f142c898b80127ca0708b6" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 33951 } } peers { permanent_uuid: "234a5283e54b46818e3522c5075446e0" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 40495 } }
I20260929 19:05:19.013602 22058 ts_tablet_manager.cc:1428] T ca1789938dba43fba9d4e3075bc5a799 P 8cf4d02c8b8b44eca96dd6b23d067157: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 19:05:19.013690 22061 tablet_bootstrap.cc:654] T ca1789938dba43fba9d4e3075bc5a799 P 7564d80b28f142c898b80127ca0708b6: Neither blocks nor log segments found. Creating new log.
I20260929 19:05:19.013720 21757 heartbeater.cc:503] Master 127.8.158.126:43397 was elected leader, sending a full tablet report...
I20260929 19:05:19.013970 22061 log.cc:824] T ca1789938dba43fba9d4e3075bc5a799 P 7564d80b28f142c898b80127ca0708b6: Log is configured to *not* fsync() on all Append() calls
I20260929 19:05:19.014758 22060 tablet_bootstrap.cc:492] T ca1789938dba43fba9d4e3075bc5a799 P 234a5283e54b46818e3522c5075446e0: No bootstrap required, opened a new log
I20260929 19:05:19.014838 22060 ts_tablet_manager.cc:1397] T ca1789938dba43fba9d4e3075bc5a799 P 234a5283e54b46818e3522c5075446e0: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 19:05:19.017081 22060 raft_consensus.cc:348] T ca1789938dba43fba9d4e3075bc5a799 P 234a5283e54b46818e3522c5075446e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf4d02c8b8b44eca96dd6b23d067157" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 35713 } } peers { permanent_uuid: "7564d80b28f142c898b80127ca0708b6" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 33951 } } peers { permanent_uuid: "234a5283e54b46818e3522c5075446e0" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 40495 } }
I20260929 19:05:19.017227 22060 raft_consensus.cc:374] T ca1789938dba43fba9d4e3075bc5a799 P 234a5283e54b46818e3522c5075446e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:05:19.017256 22060 raft_consensus.cc:729] T ca1789938dba43fba9d4e3075bc5a799 P 234a5283e54b46818e3522c5075446e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 234a5283e54b46818e3522c5075446e0, State: Initialized, Role: FOLLOWER
I20260929 19:05:19.017350 22060 consensus_queue.cc:260] T ca1789938dba43fba9d4e3075bc5a799 P 234a5283e54b46818e3522c5075446e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf4d02c8b8b44eca96dd6b23d067157" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 35713 } } peers { permanent_uuid: "7564d80b28f142c898b80127ca0708b6" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 33951 } } peers { permanent_uuid: "234a5283e54b46818e3522c5075446e0" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 40495 } }
I20260929 19:05:19.017575 22060 ts_tablet_manager.cc:1428] T ca1789938dba43fba9d4e3075bc5a799 P 234a5283e54b46818e3522c5075446e0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 19:05:19.017620 22025 heartbeater.cc:503] Master 127.8.158.126:43397 was elected leader, sending a full tablet report...
I20260929 19:05:19.018155 22061 tablet_bootstrap.cc:492] T ca1789938dba43fba9d4e3075bc5a799 P 7564d80b28f142c898b80127ca0708b6: No bootstrap required, opened a new log
I20260929 19:05:19.018214 22061 ts_tablet_manager.cc:1397] T ca1789938dba43fba9d4e3075bc5a799 P 7564d80b28f142c898b80127ca0708b6: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260929 19:05:19.020532 22061 raft_consensus.cc:348] T ca1789938dba43fba9d4e3075bc5a799 P 7564d80b28f142c898b80127ca0708b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf4d02c8b8b44eca96dd6b23d067157" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 35713 } } peers { permanent_uuid: "7564d80b28f142c898b80127ca0708b6" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 33951 } } peers { permanent_uuid: "234a5283e54b46818e3522c5075446e0" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 40495 } }
I20260929 19:05:19.020682 22061 raft_consensus.cc:374] T ca1789938dba43fba9d4e3075bc5a799 P 7564d80b28f142c898b80127ca0708b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:05:19.020716 22061 raft_consensus.cc:729] T ca1789938dba43fba9d4e3075bc5a799 P 7564d80b28f142c898b80127ca0708b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7564d80b28f142c898b80127ca0708b6, State: Initialized, Role: FOLLOWER
I20260929 19:05:19.020828 22061 consensus_queue.cc:260] T ca1789938dba43fba9d4e3075bc5a799 P 7564d80b28f142c898b80127ca0708b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf4d02c8b8b44eca96dd6b23d067157" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 35713 } } peers { permanent_uuid: "7564d80b28f142c898b80127ca0708b6" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 33951 } } peers { permanent_uuid: "234a5283e54b46818e3522c5075446e0" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 40495 } }
I20260929 19:05:19.021054 22061 ts_tablet_manager.cc:1428] T ca1789938dba43fba9d4e3075bc5a799 P 7564d80b28f142c898b80127ca0708b6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 19:05:19.021153 21891 heartbeater.cc:503] Master 127.8.158.126:43397 was elected leader, sending a full tablet report...
W20260929 19:05:19.077652 21758 tablet.cc:2369] T ca1789938dba43fba9d4e3075bc5a799 P 8cf4d02c8b8b44eca96dd6b23d067157: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 19:05:19 dist-test-slave-nmsz krb5kdc[20563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708718, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for HTTP/127.8.158.84@KRBTEST.COM
W20260929 19:05:19.211206 22026 tablet.cc:2369] T ca1789938dba43fba9d4e3075bc5a799 P 234a5283e54b46818e3522c5075446e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 19:05:19.268657 21892 tablet.cc:2369] T ca1789938dba43fba9d4e3075bc5a799 P 7564d80b28f142c898b80127ca0708b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:05:19.284850 22066 raft_consensus.cc:482] T ca1789938dba43fba9d4e3075bc5a799 P 7564d80b28f142c898b80127ca0708b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:05:19.284989 22066 raft_consensus.cc:504] T ca1789938dba43fba9d4e3075bc5a799 P 7564d80b28f142c898b80127ca0708b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf4d02c8b8b44eca96dd6b23d067157" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 35713 } } peers { permanent_uuid: "7564d80b28f142c898b80127ca0708b6" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 33951 } } peers { permanent_uuid: "234a5283e54b46818e3522c5075446e0" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 40495 } }
I20260929 19:05:19.285393 22066 leader_election.cc:288] T ca1789938dba43fba9d4e3075bc5a799 P 7564d80b28f142c898b80127ca0708b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8cf4d02c8b8b44eca96dd6b23d067157 (127.8.158.65:35713), 234a5283e54b46818e3522c5075446e0 (127.8.158.67:40495)
I20260929 19:05:19.289808 22055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:19.285794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:57135 (local address 127.8.158.65:35713)
0929 19:05:19.285838 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:19.285840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:19.286334 (+ 494us) server_negotiation.cc:407] Connection header received
0929 19:05:19.286437 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:19.286438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:19.286448 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:19.286477 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:05:19.286707 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:19.287447 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:19.288862 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:19.289414 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:19.289462 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:19.289470 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:19.289697 (+ 227us) server_negotiation.cc:299] Negotiation successful
0929 19:05:19.289735 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 19:05:19.290051 22069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:19.285605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:35713 (local address 127.8.158.66:57135)
0929 19:05:19.286253 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:19.286264 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:19.286362 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:19.286550 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:19.286550 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:19.286552 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:05:19.286627 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:19.286629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:19.287549 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:19.287550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:19.288759 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:19.288762 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:19.289544 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:19.289545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:19.289614 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:19.289618 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:05:19.289925 (+ 307us) client_negotiation.cc:240] Negotiation successful
0929 19:05:19.289965 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":587,"threads_started":1}
I20260929 19:05:19.290416 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca1789938dba43fba9d4e3075bc5a799" candidate_uuid: "7564d80b28f142c898b80127ca0708b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cf4d02c8b8b44eca96dd6b23d067157" is_pre_election: true
I20260929 19:05:19.290596 21698 raft_consensus.cc:2454] T ca1789938dba43fba9d4e3075bc5a799 P 8cf4d02c8b8b44eca96dd6b23d067157 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7564d80b28f142c898b80127ca0708b6 in term 0.
I20260929 19:05:19.290838 21780 leader_election.cc:302] T ca1789938dba43fba9d4e3075bc5a799 P 7564d80b28f142c898b80127ca0708b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7564d80b28f142c898b80127ca0708b6, 8cf4d02c8b8b44eca96dd6b23d067157; no voters:
I20260929 19:05:19.291466 22066 raft_consensus.cc:2781] T ca1789938dba43fba9d4e3075bc5a799 P 7564d80b28f142c898b80127ca0708b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:05:19.291543 22066 raft_consensus.cc:482] T ca1789938dba43fba9d4e3075bc5a799 P 7564d80b28f142c898b80127ca0708b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:05:19.291574 22066 raft_consensus.cc:3037] T ca1789938dba43fba9d4e3075bc5a799 P 7564d80b28f142c898b80127ca0708b6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:05:19.292470 22066 raft_consensus.cc:504] T ca1789938dba43fba9d4e3075bc5a799 P 7564d80b28f142c898b80127ca0708b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf4d02c8b8b44eca96dd6b23d067157" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 35713 } } peers { permanent_uuid: "7564d80b28f142c898b80127ca0708b6" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 33951 } } peers { permanent_uuid: "234a5283e54b46818e3522c5075446e0" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 40495 } }
I20260929 19:05:19.292609 22066 leader_election.cc:288] T ca1789938dba43fba9d4e3075bc5a799 P 7564d80b28f142c898b80127ca0708b6 [CANDIDATE]: Term 1 election: Requested vote from peers 8cf4d02c8b8b44eca96dd6b23d067157 (127.8.158.65:35713), 234a5283e54b46818e3522c5075446e0 (127.8.158.67:40495)
I20260929 19:05:19.292796 21698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca1789938dba43fba9d4e3075bc5a799" candidate_uuid: "7564d80b28f142c898b80127ca0708b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cf4d02c8b8b44eca96dd6b23d067157"
I20260929 19:05:19.292879 21698 raft_consensus.cc:3037] T ca1789938dba43fba9d4e3075bc5a799 P 8cf4d02c8b8b44eca96dd6b23d067157 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:05:19.293736 21698 raft_consensus.cc:2454] T ca1789938dba43fba9d4e3075bc5a799 P 8cf4d02c8b8b44eca96dd6b23d067157 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7564d80b28f142c898b80127ca0708b6 in term 1.
I20260929 19:05:19.293908 21780 leader_election.cc:302] T ca1789938dba43fba9d4e3075bc5a799 P 7564d80b28f142c898b80127ca0708b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7564d80b28f142c898b80127ca0708b6, 8cf4d02c8b8b44eca96dd6b23d067157; no voters:
I20260929 19:05:19.293984 22066 raft_consensus.cc:2781] T ca1789938dba43fba9d4e3075bc5a799 P 7564d80b28f142c898b80127ca0708b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:05:19.294111 22066 raft_consensus.cc:686] T ca1789938dba43fba9d4e3075bc5a799 P 7564d80b28f142c898b80127ca0708b6 [term 1 LEADER]: Becoming Leader. State: Replica: 7564d80b28f142c898b80127ca0708b6, State: Running, Role: LEADER
I20260929 19:05:19.294211 22066 consensus_queue.cc:237] T ca1789938dba43fba9d4e3075bc5a799 P 7564d80b28f142c898b80127ca0708b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf4d02c8b8b44eca96dd6b23d067157" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 35713 } } peers { permanent_uuid: "7564d80b28f142c898b80127ca0708b6" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 33951 } } peers { permanent_uuid: "234a5283e54b46818e3522c5075446e0" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 40495 } }
I20260929 19:05:19.295025 21559 catalog_manager.cc:5495] T ca1789938dba43fba9d4e3075bc5a799 P 7564d80b28f142c898b80127ca0708b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7564d80b28f142c898b80127ca0708b6 (127.8.158.66). New cstate: current_term: 1 leader_uuid: "7564d80b28f142c898b80127ca0708b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cf4d02c8b8b44eca96dd6b23d067157" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 35713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7564d80b28f142c898b80127ca0708b6" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 33951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "234a5283e54b46818e3522c5075446e0" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 40495 } health_report { overall_health: UNKNOWN } } }
I20260929 19:05:19.298597 22056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:19.286096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:44983 (local address 127.8.158.67:40495)
0929 19:05:19.286140 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:19.286141 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:19.290344 (+ 4203us) server_negotiation.cc:407] Connection header received
0929 19:05:19.290448 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:19.290449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:19.290459 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:19.290490 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:05:19.290652 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:19.291353 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:19.297612 (+ 6259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:19.298160 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:19.298228 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:19.298237 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:19.298485 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 19:05:19.298531 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 19:05:19.298825 22069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:19.285919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.67:40495 (local address 127.8.158.66:44983)
0929 19:05:19.290269 (+ 4350us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:19.290279 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:19.290371 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:19.290554 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:19.290555 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:19.290556 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:05:19.290583 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:19.290584 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:19.296463 (+ 5879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:19.296464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:19.297499 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:19.297502 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:19.298294 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:19.298295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:19.298389 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:19.298393 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:05:19.298723 (+ 330us) client_negotiation.cc:240] Negotiation successful
0929 19:05:19.298762 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4339,"thread_start_us":89,"threads_started":1}
I20260929 19:05:19.299135 21963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca1789938dba43fba9d4e3075bc5a799" candidate_uuid: "7564d80b28f142c898b80127ca0708b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "234a5283e54b46818e3522c5075446e0" is_pre_election: true
I20260929 19:05:19.299189 21964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca1789938dba43fba9d4e3075bc5a799" candidate_uuid: "7564d80b28f142c898b80127ca0708b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "234a5283e54b46818e3522c5075446e0"
I20260929 19:05:19.299284 21963 raft_consensus.cc:2454] T ca1789938dba43fba9d4e3075bc5a799 P 234a5283e54b46818e3522c5075446e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7564d80b28f142c898b80127ca0708b6 in term 0.
I20260929 19:05:19.356096 22057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:19.353513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32970 (local address 127.8.158.66:33951)
0929 19:05:19.353570 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:19.353572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:19.353579 (+ 7us) server_negotiation.cc:407] Connection header received
0929 19:05:19.353624 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:19.353624 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:19.353640 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:19.353720 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:05:19.353869 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:19.354434 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:19.355270 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:19.355488 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:19.355575 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:19.355746 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:05:19.355853 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:05:19.355949 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:19.355973 (+ 24us) server_negotiation.cc:299] Negotiation successful
0929 19:05:19.355993 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 19:05:19.358057 21963 raft_consensus.cc:3037] T ca1789938dba43fba9d4e3075bc5a799 P 234a5283e54b46818e3522c5075446e0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:05:19.358137 21698 raft_consensus.cc:1261] T ca1789938dba43fba9d4e3075bc5a799 P 8cf4d02c8b8b44eca96dd6b23d067157 [term 1 FOLLOWER]: Refusing update from remote peer 7564d80b28f142c898b80127ca0708b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:05:19.358489 22066 consensus_queue.cc:1035] T ca1789938dba43fba9d4e3075bc5a799 P 7564d80b28f142c898b80127ca0708b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cf4d02c8b8b44eca96dd6b23d067157" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 35713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:05:19.359119 21963 raft_consensus.cc:1261] T ca1789938dba43fba9d4e3075bc5a799 P 234a5283e54b46818e3522c5075446e0 [term 1 FOLLOWER]: Refusing update from remote peer 7564d80b28f142c898b80127ca0708b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:05:19.360163 22066 consensus_queue.cc:1035] T ca1789938dba43fba9d4e3075bc5a799 P 7564d80b28f142c898b80127ca0708b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "234a5283e54b46818e3522c5075446e0" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 40495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:05:19.362399 22081 mvcc.cc:205] Tried to move back new op lower bound from 7334742914488119296 to 7334742914229837824. Current Snapshot: MvccSnapshot[applied={T|T < 7334742914488119296 or (T in {7334742914488119296})}]
I20260929 19:05:19.363233 22082 mvcc.cc:205] Tried to move back new op lower bound from 7334742914488119296 to 7334742914229837824. Current Snapshot: MvccSnapshot[applied={T|T < 7334742914488119296}]
I20260929 19:05:19.363561 21558 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:05:19.363651 21558 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:05:19.364401 21558 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5b0248c724d24afd92a1a9d821b2988e: Sending DeleteTablet for 3 replicas of tablet ca1789938dba43fba9d4e3075bc5a799
I20260929 19:05:19.365026 21825 tablet_service.cc:1514] Processing DeleteTablet for tablet ca1789938dba43fba9d4e3075bc5a799 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:05:19 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:32964
I20260929 19:05:19.365083 21673 tablet_service.cc:1514] Processing DeleteTablet for tablet ca1789938dba43fba9d4e3075bc5a799 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:05:19 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:59466
I20260929 19:05:19.365065 21959 tablet_service.cc:1514] Processing DeleteTablet for tablet ca1789938dba43fba9d4e3075bc5a799 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:05:19 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:50204
I20260929 19:05:19.365762 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 21597
I20260929 19:05:19.365823 22085 mvcc.cc:205] Tried to move back new op lower bound from 7334742914488119296 to 7334742914229837824. Current Snapshot: MvccSnapshot[applied={T|T < 7334742914488119296}]
I20260929 19:05:19.374079 22090 tablet_replica.cc:330] T ca1789938dba43fba9d4e3075bc5a799 P 234a5283e54b46818e3522c5075446e0: stopping tablet replica
I20260929 19:05:19.374260 22090 raft_consensus.cc:2229] T ca1789938dba43fba9d4e3075bc5a799 P 234a5283e54b46818e3522c5075446e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:05:19.374454 22090 raft_consensus.cc:2258] T ca1789938dba43fba9d4e3075bc5a799 P 234a5283e54b46818e3522c5075446e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:05:19.374668 22089 tablet_replica.cc:330] T ca1789938dba43fba9d4e3075bc5a799 P 7564d80b28f142c898b80127ca0708b6: stopping tablet replica
I20260929 19:05:19.374832 22089 raft_consensus.cc:2229] T ca1789938dba43fba9d4e3075bc5a799 P 7564d80b28f142c898b80127ca0708b6 [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:05:19.375152 22090 ts_tablet_manager.cc:1905] T ca1789938dba43fba9d4e3075bc5a799 P 234a5283e54b46818e3522c5075446e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:05:19.376857 22089 raft_consensus.cc:2258] T ca1789938dba43fba9d4e3075bc5a799 P 7564d80b28f142c898b80127ca0708b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:05:19.377554 22089 ts_tablet_manager.cc:1905] T ca1789938dba43fba9d4e3075bc5a799 P 7564d80b28f142c898b80127ca0708b6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 19:05:19.377813 21545 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.158.65:35713 (error 108)
I20260929 19:05:19.378424 22090 ts_tablet_manager.cc:1918] T ca1789938dba43fba9d4e3075bc5a799 P 234a5283e54b46818e3522c5075446e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:05:19.378695 22090 log.cc:1196] T ca1789938dba43fba9d4e3075bc5a799 P 234a5283e54b46818e3522c5075446e0: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708573736807-8825-0/minicluster-data/ts-2/wal/wals/ca1789938dba43fba9d4e3075bc5a799
I20260929 19:05:19.380484 22089 ts_tablet_manager.cc:1918] T ca1789938dba43fba9d4e3075bc5a799 P 7564d80b28f142c898b80127ca0708b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:05:19.380694 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 21761
I20260929 19:05:19.381744 22090 ts_tablet_manager.cc:1939] T ca1789938dba43fba9d4e3075bc5a799 P 234a5283e54b46818e3522c5075446e0: Deleting consensus metadata
I20260929 19:05:19.382208 21543 catalog_manager.cc:4841] TS 234a5283e54b46818e3522c5075446e0 (127.8.158.67:40495): tablet ca1789938dba43fba9d4e3075bc5a799 (table test-table [id=5fddcccc7ddb4ef099ac8c537e9df41c]) successfully deleted
I20260929 19:05:19.386088 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 21894
I20260929 19:05:19.390691 22052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:19.390559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:35713 (local address 127.0.0.1:59480)
0929 19:05:19.390603 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:19.390646 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.65:35713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260929 19:05:19.390802 22053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:19.390712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.66:33951 (local address 127.0.0.1:32986)
0929 19:05:19.390761 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:19.390783 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.66:33951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 19:05:19.390829 21545 catalog_manager.cc:4584] TS 8cf4d02c8b8b44eca96dd6b23d067157 (127.8.158.65:35713): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ca1789938dba43fba9d4e3075bc5a799: Network error: Client connection negotiation failed: client connection to 127.8.158.65:35713: connect: Connection refused (error 111)
I20260929 19:05:19.392881 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 21526
I20260929 19:05:19.399324 8825 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 19:05:19.743667 8825 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 19:05:19.743777 8825 mini_ranger.cc:66] Stopping Ranger...
I20260929 19:05:20.087718 8825 mini_ranger.cc:68] Stopped Ranger
2026-09-29 19:05:20.087 UTC [20593] LOG: received smart shutdown request
2026-09-29 19:05:20.091 UTC [20593] LOG: background worker "logical replication launcher" (PID 20602) exited with exit code 1
2026-09-29 19:05:20.091 UTC [20597] LOG: shutting down
2026-09-29 19:05:20.269 UTC [20593] LOG: database system is shut down
2026-09-29T19:05:20Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (46668 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:05:20 dist-test-slave-nmsz krb5kdc[22103](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:05:20 dist-test-slave-nmsz krb5kdc[22103](info): set up 2 sockets
Sep 29 19:05:20 dist-test-slave-nmsz krb5kdc[22103](info): commencing operation
krb5kdc: starting...
W20260929 19:05:21.418080 8825 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Sep 29 19:05:21 dist-test-slave-nmsz krb5kdc[22103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:05:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:05:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.158.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
I20260929 19:05:21.529848 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:35161
--webserver_interface=127.8.158.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:45677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:35161,127.8.158.125:36407,127.8.158.124:46399
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:05:21.607502 22116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:05:21.607702 22116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:05:21.607724 22116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:05:21.608781 22116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:05:21.608819 22116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:05:21.608834 22116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:05:21.608848 22116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:05:21.608860 22116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:05:21.610260 22116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:45677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:35161,127.8.158.125:36407,127.8.158.124:46399
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:35161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.158.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:05:21.610581 22116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:05:21.610790 22116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:05:21.615651 22122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:21.615770 22124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:21.615698 22121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:05:21.616071 22116 server_base.cc:1034] running on GCE node
I20260929 19:05:21.616317 22116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:05:21.616711 22116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:05:21.617901 22116 hybrid_clock.cc:648] HybridClock initialized: now 1790708721617869 us; error 43 us; skew 500 ppm
Sep 29 19:05:21 dist-test-slave-nmsz krb5kdc[22103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708721, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:05:21.620581 22116 init.cc:377] Logged in from keytab as kudu/127.8.158.126@KRBTEST.COM (short username kudu)
I20260929 19:05:21.621181 22116 webserver.cc:460] Webserver started at http://127.8.158.126:42103/ using document root <none> and password file <none>
I20260929 19:05:21.621435 22116 fs_manager.cc:359] Metadata directory not provided
I20260929 19:05:21.621513 22116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:05:21.621675 22116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:05:21.622879 22116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "70fe724446634be9b867efe7c51c76f4"
format_stamp: "Formatted at 2026-09-29 19:05:21 on dist-test-slave-nmsz"
server_key: "281e94fd3e4d834511fb88d687cd2e29"
server_key_iv: "da681f0be16f80e2efcb57fa2e814573"
server_key_version: "encryptionkey@0"
I20260929 19:05:21.623293 22116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "70fe724446634be9b867efe7c51c76f4"
format_stamp: "Formatted at 2026-09-29 19:05:21 on dist-test-slave-nmsz"
server_key: "281e94fd3e4d834511fb88d687cd2e29"
server_key_iv: "da681f0be16f80e2efcb57fa2e814573"
server_key_version: "encryptionkey@0"
I20260929 19:05:21.624730 22116 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:05:21.625481 22131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:21.625664 22116 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:05:21.625761 22116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "70fe724446634be9b867efe7c51c76f4"
format_stamp: "Formatted at 2026-09-29 19:05:21 on dist-test-slave-nmsz"
server_key: "281e94fd3e4d834511fb88d687cd2e29"
server_key_iv: "da681f0be16f80e2efcb57fa2e814573"
server_key_version: "encryptionkey@0"
I20260929 19:05:21.625842 22116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:05:21.662626 22116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:05:21.665002 22116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:05:21.665300 22116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:05:21.669656 22116 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.126:35161
I20260929 19:05:21.669684 22183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.126:35161 every 8 connection(s)
I20260929 19:05:21.670073 22116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:05:21.671036 22184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:05:21.672515 22184 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35161 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:05:21.674305 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 22116
I20260929 19:05:21.674470 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:05:21.674647 8825 external_mini_cluster.cc:1428] Setting key 0234bed71467a96f3bd1a2fcade70403
Sep 29 19:05:21 dist-test-slave-nmsz krb5kdc[22103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708721, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.158.125@KRBTEST.COM; defaulting to no policy
I20260929 19:05:21.682987 22186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:21.673087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35161 (local address 127.0.0.1:56922)
0929 19:05:21.673316 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:21.673358 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:21.673850 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:21.674143 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:21.674145 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:21.674560 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:05:21.674765 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:21.674771 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:21.675823 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:21.675824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:21.676492 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:21.676494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:21.677033 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:21.677034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:21.677119 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:21.677585 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:05:21.677594 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:05:21.679468 (+ 1874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:05:21.681477 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:21.681483 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:21.681484 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:21.681671 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:21.681976 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:21.681977 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:21.681977 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:21.682058 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:21.682525 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:05:21.682530 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:05:21.682586 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 19:05:21.682674 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 19:05:21.682790 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":89,"threads_started":1}
I20260929 19:05:21.682998 22187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:21.673161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56922 (local address 127.8.158.126:35161)
0929 19:05:21.673325 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:21.673334 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:21.673456 (+ 122us) server_negotiation.cc:407] Connection header received
0929 19:05:21.674013 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:21.674015 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:21.674041 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:21.674154 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:21.674865 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:21.675709 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:21.676582 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:21.676943 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:21.677022 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:21.679580 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:21.679588 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:21.679589 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:21.679596 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:21.681394 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:21.681763 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:21.681766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:21.681766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:21.681869 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:21.682130 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:21.682131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:21.682131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:21.682451 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:21.682544 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:21.682719 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 19:05:21.682847 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":124,"thread_start_us":67,"threads_started":1}
I20260929 19:05:21.683766 22184 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.158.125" port: 36407 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:05:21.684154 22186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:21.684009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.125:36407 (local address 127.0.0.1:41964)
0929 19:05:21.684086 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:21.684134 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.125:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260929 19:05:21.684401 22135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.158.125:36407: connect: Connection refused (error 111)
I20260929 19:05:21.684979 22186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:21.684844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.125:36407 (local address 127.0.0.1:41972)
0929 19:05:21.684898 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:21.684949 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.125:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.125@KRBTEST.COM" created.
W20260929 19:05:21.685087 22184 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.158.125:36407: Network error: Client connection negotiation failed: client connection to 127.8.158.125:36407: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.125.keytab.
Entry for principal kudu/127.8.158.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.125.keytab.
I20260929 19:05:21.699123 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.125:36407
--webserver_interface=127.8.158.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.125.keytab
--principal=kudu/127.8.158.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:45677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:35161,127.8.158.125:36407,127.8.158.124:46399
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:05:21.734332 22184 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.158.125" port: 36407 } attempt: 1
I20260929 19:05:21.734824 22186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:21.734681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.125:36407 (local address 127.0.0.1:41980)
0929 19:05:21.734743 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:21.734789 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.125:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 19:05:21.735270 22186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:21.735171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.125:36407 (local address 127.0.0.1:41996)
0929 19:05:21.735215 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:21.735248 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.125:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260929 19:05:21.735368 22184 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.158.125:36407: Network error: Client connection negotiation failed: client connection to 127.8.158.125:36407: connect: Connection refused (error 111)
W20260929 19:05:21.778491 22191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:05:21.778666 22191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:05:21.778687 22191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:05:21.779713 22191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:05:21.779753 22191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:05:21.779767 22191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:05:21.779780 22191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:05:21.779793 22191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:05:21.781152 22191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:45677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:35161,127.8.158.125:36407,127.8.158.124:46399
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.125.keytab
--principal=kudu/127.8.158.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.125:36407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.158.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:05:21.781320 22191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:05:21.781469 22191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 19:05:21.786206 22191 server_base.cc:1034] running on GCE node
W20260929 19:05:21.786111 22196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:21.786111 22199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:21.786124 22197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:05:21.786576 22191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:05:21.786863 22191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:05:21.788033 22191 hybrid_clock.cc:648] HybridClock initialized: now 1790708721788017 us; error 26 us; skew 500 ppm
Sep 29 19:05:21 dist-test-slave-nmsz krb5kdc[22103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708721, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:05:21.790539 22191 init.cc:377] Logged in from keytab as kudu/127.8.158.125@KRBTEST.COM (short username kudu)
I20260929 19:05:21.790966 22191 webserver.cc:460] Webserver started at http://127.8.158.125:46403/ using document root <none> and password file <none>
I20260929 19:05:21.791196 22191 fs_manager.cc:359] Metadata directory not provided
I20260929 19:05:21.791266 22191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:05:21.791393 22191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:05:21.792362 22191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-1/data/instance:
uuid: "0712bc27ee6340ad85fab7ab0927cc3c"
format_stamp: "Formatted at 2026-09-29 19:05:21 on dist-test-slave-nmsz"
server_key: "573f188b374a4e9e33bbd10abaa43d1a"
server_key_iv: "dc3c94b4a021c95da92b6e5532443114"
server_key_version: "encryptionkey@0"
I20260929 19:05:21.792797 22191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-1/wal/instance:
uuid: "0712bc27ee6340ad85fab7ab0927cc3c"
format_stamp: "Formatted at 2026-09-29 19:05:21 on dist-test-slave-nmsz"
server_key: "573f188b374a4e9e33bbd10abaa43d1a"
server_key_iv: "dc3c94b4a021c95da92b6e5532443114"
server_key_version: "encryptionkey@0"
I20260929 19:05:21.794034 22191 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:05:21.794808 22206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:21.794978 22191 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:21.795066 22191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-1/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-1/wal
uuid: "0712bc27ee6340ad85fab7ab0927cc3c"
format_stamp: "Formatted at 2026-09-29 19:05:21 on dist-test-slave-nmsz"
server_key: "573f188b374a4e9e33bbd10abaa43d1a"
server_key_iv: "dc3c94b4a021c95da92b6e5532443114"
server_key_version: "encryptionkey@0"
I20260929 19:05:21.795151 22191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:05:21.803490 22184 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.158.125" port: 36407 } attempt: 2
I20260929 19:05:21.803836 22186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:21.803716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.125:36407 (local address 127.0.0.1:42008)
0929 19:05:21.803774 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:21.803808 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.125:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 19:05:21.803930 22191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:05:21.804186 22186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:21.804066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.125:36407 (local address 127.0.0.1:42016)
0929 19:05:21.804109 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:21.804155 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.125:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 19:05:21.804352 22184 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.158.125:36407: Network error: Client connection negotiation failed: client connection to 127.8.158.125:36407: connect: Connection refused (error 111)
I20260929 19:05:21.805759 22191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:05:21.805915 22191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:05:21.809542 22191 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.125:36407
I20260929 19:05:21.809574 22258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.125:36407 every 8 connection(s)
I20260929 19:05:21.809885 22191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-1/data/info.pb
I20260929 19:05:21.810739 22259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:05:21.812178 22259 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35161 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:05:21.813127 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 22191
I20260929 19:05:21.813251 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-1/wal/instance
I20260929 19:05:21.813434 8825 external_mini_cluster.cc:1428] Setting key 7d1532a11d6064b41991fb20908e1730
Sep 29 19:05:21 dist-test-slave-nmsz krb5kdc[22103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708721, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.125@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.158.124@KRBTEST.COM; defaulting to no policy
I20260929 19:05:21.821578 22187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:21.812826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56934 (local address 127.8.158.126:35161)
0929 19:05:21.812897 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:21.812899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:21.813124 (+ 225us) server_negotiation.cc:407] Connection header received
0929 19:05:21.813624 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:21.813626 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:21.813638 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:21.813709 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:21.814256 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:21.814990 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:21.816248 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:21.816600 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:21.816683 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:21.818698 (+ 2015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:21.818706 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:21.818706 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:21.818714 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:21.820247 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:21.820613 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:21.820615 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:21.820615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:21.820675 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:21.820914 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:21.820915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:21.820916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:21.821044 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:21.821117 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:21.821390 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 19:05:21.821480 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 19:05:21.821702 22261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:21.812792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35161 (local address 127.0.0.1:56934)
0929 19:05:21.812984 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:21.813021 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:21.813519 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:21.813775 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:21.813777 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:21.814000 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:05:21.814172 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:21.814175 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:21.815078 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:21.815080 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:21.816146 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:21.816148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:21.816684 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:21.816685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:21.816792 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:21.817281 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:05:21.817296 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:05:21.818609 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:05:21.820328 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:21.820333 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:21.820334 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:21.820535 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:21.820761 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:21.820762 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:21.820762 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:21.820848 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:21.821110 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:05:21.821114 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:05:21.821265 (+ 151us) client_negotiation.cc:769] Sending connection context
0929 19:05:21.821362 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 19:05:21.821519 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":75,"threads_started":1}
I20260929 19:05:21.822243 22259 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.158.125" port: 36407 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.124@KRBTEST.COM" created.
Sep 29 19:05:21 dist-test-slave-nmsz krb5kdc[22103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708721, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.125@KRBTEST.COM for kudu/127.8.158.125@KRBTEST.COM
I20260929 19:05:21.830257 22261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:21.822509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.125:36407 (local address 127.0.0.1:42032)
0929 19:05:21.822556 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:21.822567 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:21.822608 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:21.822818 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:21.822818 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:21.822950 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:05:21.822986 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:21.822987 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:21.823982 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:21.823983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:21.824507 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:21.824508 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:21.825001 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:21.825003 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:21.825090 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:21.825516 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:05:21.825521 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:05:21.826704 (+ 1183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:05:21.829090 (+ 2386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:21.829091 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:21.829091 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:21.829335 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:21.829579 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:21.829580 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:21.829580 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:21.829638 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:21.830024 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:05:21.830025 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:05:21.830062 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 19:05:21.830125 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 19:05:21.830196 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 19:05:21.830408 22263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:21.822563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42032 (local address 127.8.158.125:36407)
0929 19:05:21.822679 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:21.822685 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:21.822704 (+ 19us) server_negotiation.cc:407] Connection header received
0929 19:05:21.822719 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:21.822720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:21.822739 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:21.822818 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:21.823074 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:21.823889 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:21.824599 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:21.824907 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:21.824978 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:21.826803 (+ 1825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:21.826809 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:21.826810 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:21.826815 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:21.828943 (+ 2128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:21.829430 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:21.829433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:21.829433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:21.829501 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:21.829713 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:21.829714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:21.829715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:21.829896 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:21.829987 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:21.830192 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 19:05:21.830275 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":64,"threads_started":1}
I20260929 19:05:21.831079 22259 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46399 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:05:21.831419 22261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:21.831309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.124:46399 (local address 127.0.0.1:57850)
0929 19:05:21.831357 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:21.831402 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.124:46399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260929 19:05:21.831544 22209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.158.124:46399: connect: Connection refused (error 111)
I20260929 19:05:21.831961 22261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:21.831872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.124:46399 (local address 127.0.0.1:57860)
0929 19:05:21.831918 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:21.831941 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.124:46399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 19:05:21.832067 22259 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.158.124:46399: Network error: Client connection negotiation failed: client connection to 127.8.158.124:46399: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.124.keytab.
Entry for principal kudu/127.8.158.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.124.keytab.
I20260929 19:05:21.840749 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.124:46399
--webserver_interface=127.8.158.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.124.keytab
--principal=kudu/127.8.158.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:45677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:35161,127.8.158.125:36407,127.8.158.124:46399
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:05:21.881332 22259 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46399 } attempt: 1
I20260929 19:05:21.881853 22261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:21.881705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.124:46399 (local address 127.0.0.1:57870)
0929 19:05:21.881774 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:21.881815 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.124:46399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 19:05:21.882269 22261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:21.882128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.124:46399 (local address 127.0.0.1:57878)
0929 19:05:21.882198 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:21.882240 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.124:46399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 19:05:21.882433 22259 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.158.124:46399: Network error: Client connection negotiation failed: client connection to 127.8.158.124:46399: connect: Connection refused (error 111)
I20260929 19:05:21.895510 22184 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.158.125" port: 36407 } attempt: 3
Sep 29 19:05:21 dist-test-slave-nmsz krb5kdc[22103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708721, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for kudu/127.8.158.125@KRBTEST.COM
I20260929 19:05:21.903357 22263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:21.895925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42044 (local address 127.8.158.125:36407)
0929 19:05:21.895984 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:21.895986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:21.896026 (+ 40us) server_negotiation.cc:407] Connection header received
0929 19:05:21.896049 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:21.896050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:21.896059 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:21.896104 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:21.896460 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:21.897001 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:21.898050 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:21.898462 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:21.898523 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:21.900570 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:21.900571 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:21.900571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:21.900580 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:21.902088 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:21.902571 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:21.902573 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:21.902573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:21.902634 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:21.902841 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:21.902842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:21.902842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:21.902959 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:21.903037 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:21.903187 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 19:05:21.903262 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 19:05:21.903366 22186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:21.895841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.125:36407 (local address 127.0.0.1:42044)
0929 19:05:21.895907 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:21.895920 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:21.895986 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:21.896126 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:21.896127 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:21.896313 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:05:21.896386 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:21.896388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:21.897110 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:21.897111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:21.897963 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:21.897966 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:21.898535 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:21.898536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:21.898615 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:21.899081 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:05:21.899086 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:05:21.900474 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:05:21.902212 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:21.902213 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:21.902213 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:21.902483 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:21.902702 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:21.902703 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:21.902704 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:21.902764 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:21.903029 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:05:21.903030 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:05:21.903075 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 19:05:21.903163 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 19:05:21.903262 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260929 19:05:21.903942 22184 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46399 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:05:21.904351 22186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:21.904227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.124:46399 (local address 127.0.0.1:57892)
0929 19:05:21.904287 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:21.904329 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.124:46399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 19:05:21.904722 22186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:21.904630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.124:46399 (local address 127.0.0.1:57894)
0929 19:05:21.904676 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:21.904700 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.124:46399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260929 19:05:21.904825 22184 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.158.124:46399: Network error: Client connection negotiation failed: client connection to 127.8.158.124:46399: connect: Connection refused (error 111)
W20260929 19:05:21.921768 22266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:05:21.921945 22266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:05:21.921967 22266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:05:21.923130 22266 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:05:21.923168 22266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:05:21.923182 22266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:05:21.923194 22266 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:05:21.923226 22266 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:05:21.924738 22266 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:45677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:35161,127.8.158.125:36407,127.8.158.124:46399
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.124.keytab
--principal=kudu/127.8.158.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.124:46399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.158.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:05:21.924970 22266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:05:21.925201 22266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:05:21.929955 22274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:05:21.930116 22266 server_base.cc:1034] running on GCE node
W20260929 19:05:21.930210 22272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:21.930266 22271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:05:21.930548 22266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:05:21.930778 22266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:05:21.931921 22266 hybrid_clock.cc:648] HybridClock initialized: now 1790708721931911 us; error 25 us; skew 500 ppm
Sep 29 19:05:21 dist-test-slave-nmsz krb5kdc[22103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708721, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:05:21.934273 22266 init.cc:377] Logged in from keytab as kudu/127.8.158.124@KRBTEST.COM (short username kudu)
I20260929 19:05:21.934677 22266 webserver.cc:460] Webserver started at http://127.8.158.124:46273/ using document root <none> and password file <none>
I20260929 19:05:21.934926 22266 fs_manager.cc:359] Metadata directory not provided
I20260929 19:05:21.934994 22266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:05:21.935134 22266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:05:21.935992 22184 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46399 } attempt: 1
I20260929 19:05:21.936136 22266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-2/data/instance:
uuid: "3c2402782da146edbcfff12e7037cf1d"
format_stamp: "Formatted at 2026-09-29 19:05:21 on dist-test-slave-nmsz"
server_key: "5362a964654bf24534b7770e6bfaf015"
server_key_iv: "987772d78c7dbdf817fe2759de76c11a"
server_key_version: "encryptionkey@0"
I20260929 19:05:21.936383 22186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:21.936256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.124:46399 (local address 127.0.0.1:57906)
0929 19:05:21.936315 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:21.936349 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.124:46399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 19:05:21.936542 22266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-2/wal/instance:
uuid: "3c2402782da146edbcfff12e7037cf1d"
format_stamp: "Formatted at 2026-09-29 19:05:21 on dist-test-slave-nmsz"
server_key: "5362a964654bf24534b7770e6bfaf015"
server_key_iv: "987772d78c7dbdf817fe2759de76c11a"
server_key_version: "encryptionkey@0"
I20260929 19:05:21.936714 22186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:21.936608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.124:46399 (local address 127.0.0.1:57908)
0929 19:05:21.936661 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:21.936685 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.124:46399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 19:05:21.936805 22184 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.158.124:46399: Network error: Client connection negotiation failed: client connection to 127.8.158.124:46399: connect: Connection refused (error 111)
I20260929 19:05:21.937875 22266 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:05:21.938606 22281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:21.938776 22266 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:21.938880 22266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-2/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-2/wal
uuid: "3c2402782da146edbcfff12e7037cf1d"
format_stamp: "Formatted at 2026-09-29 19:05:21 on dist-test-slave-nmsz"
server_key: "5362a964654bf24534b7770e6bfaf015"
server_key_iv: "987772d78c7dbdf817fe2759de76c11a"
server_key_version: "encryptionkey@0"
I20260929 19:05:21.938963 22266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:05:21.950598 22259 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46399 } attempt: 2
I20260929 19:05:21.950975 22261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:21.950849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.124:46399 (local address 127.0.0.1:57920)
0929 19:05:21.950908 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:21.950942 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.124:46399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 19:05:21.951341 22261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:21.951225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.124:46399 (local address 127.0.0.1:57930)
0929 19:05:21.951290 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:21.951320 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.124:46399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 19:05:21.951517 22259 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.158.124:46399: Network error: Client connection negotiation failed: client connection to 127.8.158.124:46399: connect: Connection refused (error 111)
I20260929 19:05:21.961457 22266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:05:21.963693 22266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:05:21.963804 22266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:05:21.967483 22266 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.124:46399
I20260929 19:05:21.967530 22333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.124:46399 every 8 connection(s)
I20260929 19:05:21.967844 22266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-2/data/info.pb
I20260929 19:05:21.968703 22334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:05:21.970120 22334 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35161 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:05:21.975387 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 22266
I20260929 19:05:21.975508 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708573736807-8825-0/minicluster-data/master-2/wal/instance
Sep 29 19:05:21 dist-test-slave-nmsz krb5kdc[22103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708721, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.124@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:05:21.975710 8825 external_mini_cluster.cc:1428] Setting key 7948834e4f61d86f1e9d5d2441d0da3f
I20260929 19:05:21.979157 22187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:21.970791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56950 (local address 127.8.158.126:35161)
0929 19:05:21.970888 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:21.970891 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:21.971069 (+ 178us) server_negotiation.cc:407] Connection header received
0929 19:05:21.971573 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:21.971575 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:21.971586 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:21.971659 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:21.972220 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:21.972698 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:21.973673 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:21.973857 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:21.973928 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:21.976108 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:21.976109 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:21.976110 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:21.976118 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:21.977605 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:21.978055 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:21.978056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:21.978057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:21.978114 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:21.978417 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:21.978418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:21.978418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:21.978540 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:21.978615 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:21.978893 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 19:05:21.979014 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 19:05:21.979233 22336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:21.970775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35161 (local address 127.0.0.1:56950)
0929 19:05:21.970948 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:21.970980 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:21.971467 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:21.971722 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:21.971725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:21.971947 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:05:21.972119 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:21.972123 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:21.972792 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:21.972793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:21.973566 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:21.973567 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:21.973945 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:21.973946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:21.974064 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:21.974542 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:05:21.974549 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:05:21.976022 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:05:21.977711 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:21.977719 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:21.977721 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:21.977950 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:21.978182 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:21.978183 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:21.978183 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:21.978295 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:21.978620 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:05:21.978627 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:05:21.978755 (+ 128us) client_negotiation.cc:769] Sending connection context
0929 19:05:21.978863 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 19:05:21.979021 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":73,"threads_started":1}
I20260929 19:05:21.980031 22334 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.158.125" port: 36407 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 19:05:21 dist-test-slave-nmsz krb5kdc[22103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.158.124@KRBTEST.COM
I20260929 19:05:21.984309 22338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:21.976068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57936 (local address 127.8.158.124:46399)
0929 19:05:21.976253 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:21.976257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:21.976280 (+ 23us) server_negotiation.cc:407] Connection header received
0929 19:05:21.976303 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:21.976305 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:21.976319 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:21.976393 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:21.976934 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:21.977581 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:21.978289 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:21.978613 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:21.978706 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:21.981325 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:21.981330 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:21.981331 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:21.981336 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:21.983181 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:21.983542 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:21.983545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:21.983545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:21.983594 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:21.983798 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:21.983799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:21.983800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:21.983966 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:21.984073 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:21.984187 (+ 114us) server_negotiation.cc:299] Negotiation successful
0929 19:05:21.984243 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":92,"threads_started":1}
Sep 29 19:05:21 dist-test-slave-nmsz krb5kdc[22103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708721, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.124@KRBTEST.COM for kudu/127.8.158.125@KRBTEST.COM
I20260929 19:05:21.987413 22336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:21.980272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.125:36407 (local address 127.0.0.1:42046)
0929 19:05:21.980386 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:21.980401 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:21.980433 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:21.980560 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:21.980560 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:21.980664 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:05:21.980703 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:21.980705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:21.981799 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:21.981801 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:21.982564 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:21.982566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:21.982935 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:21.982936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:21.983013 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:21.983408 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:05:21.983412 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:05:21.984732 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:05:21.986579 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:21.986580 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:21.986581 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:21.986745 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:21.986940 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:21.986940 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:21.986940 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:21.986981 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:21.987192 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:05:21.987192 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:05:21.987220 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 19:05:21.987263 (+ 43us) client_negotiation.cc:240] Negotiation successful
0929 19:05:21.987320 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260929 19:05:21.987476 22263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:21.980343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42046 (local address 127.8.158.125:36407)
0929 19:05:21.980418 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:21.980421 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:21.980430 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:05:21.980489 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:21.980490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:21.980499 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:21.980561 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:21.980770 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:21.981455 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:21.982675 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:21.982856 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:21.982915 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:21.984956 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:21.984957 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:21.984957 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:21.984965 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:21.986506 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:21.986815 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:21.986816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:21.986816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:21.986878 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:21.987016 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:21.987017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:21.987017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:21.987140 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:21.987202 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:21.987285 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 19:05:21.987361 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 19:05:21.987838 22334 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46399 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 19:05:21 dist-test-slave-nmsz krb5kdc[22103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
Sep 29 19:05:21 dist-test-slave-nmsz krb5kdc[22103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708721, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.124@KRBTEST.COM for kudu/127.8.158.124@KRBTEST.COM
I20260929 19:05:21.992586 22187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:21.985257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56958 (local address 127.8.158.126:35161)
0929 19:05:21.985327 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:21.985329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:21.985336 (+ 7us) server_negotiation.cc:407] Connection header received
0929 19:05:21.985356 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:21.985358 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:21.985367 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:21.985424 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:21.986034 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:21.986512 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:21.987393 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:21.987673 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:21.987736 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:21.989790 (+ 2054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:21.989791 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:21.989791 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:21.989797 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:21.991460 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:21.991812 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:21.991813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:21.991813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:21.991871 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:21.992070 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:21.992070 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:21.992071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:21.992202 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:21.992373 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:21.992418 (+ 45us) server_negotiation.cc:299] Negotiation successful
0929 19:05:21.992473 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 19:05:21.994819 22336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:21.988060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.124:46399 (local address 127.0.0.1:57940)
0929 19:05:21.988085 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:21.988093 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:21.988149 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:21.988272 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:21.988272 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:21.988398 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:05:21.988419 (+ 21us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:21.988420 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:21.989041 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:21.989042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:21.989567 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:21.989569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:21.990351 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:21.990353 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:21.990415 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:21.990726 (+ 311us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:05:21.990730 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:05:21.992073 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:05:21.993827 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:21.993828 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:21.993829 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:21.994046 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:21.994275 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:21.994275 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:21.994276 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:21.994382 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:21.994618 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:05:21.994618 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:05:21.994649 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 19:05:21.994698 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 19:05:21.994765 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14}
I20260929 19:05:21.994928 22338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:21.988092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57940 (local address 127.8.158.124:46399)
0929 19:05:21.988143 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:21.988145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:21.988152 (+ 7us) server_negotiation.cc:407] Connection header received
0929 19:05:21.988206 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:21.988207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:21.988216 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:21.988268 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:21.988489 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:21.988957 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:21.989643 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:21.989923 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:21.989969 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:21.992153 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:21.992154 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:21.992154 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:21.992160 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:21.993756 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:21.994137 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:21.994138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:21.994138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:21.994197 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:21.994433 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:21.994433 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:21.994434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:21.994551 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:21.994621 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:21.994722 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 19:05:21.994824 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":35}
I20260929 19:05:21.996286 22334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c2402782da146edbcfff12e7037cf1d: Bootstrap starting.
I20260929 19:05:21.997130 22334 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c2402782da146edbcfff12e7037cf1d: Neither blocks nor log segments found. Creating new log.
I20260929 19:05:21.997448 22334 log.cc:824] T 00000000000000000000000000000000 P 3c2402782da146edbcfff12e7037cf1d: Log is configured to *not* fsync() on all Append() calls
Sep 29 19:05:21 dist-test-slave-nmsz krb5kdc[22103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.158.125@KRBTEST.COM
I20260929 19:05:21.998234 22334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c2402782da146edbcfff12e7037cf1d: No bootstrap required, opened a new log
I20260929 19:05:22.000089 22334 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c2402782da146edbcfff12e7037cf1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70fe724446634be9b867efe7c51c76f4" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35161 } } peers { permanent_uuid: "0712bc27ee6340ad85fab7ab0927cc3c" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 36407 } } peers { permanent_uuid: "3c2402782da146edbcfff12e7037cf1d" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46399 } }
I20260929 19:05:22.000255 22334 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c2402782da146edbcfff12e7037cf1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:05:22.000336 22334 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c2402782da146edbcfff12e7037cf1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c2402782da146edbcfff12e7037cf1d, State: Initialized, Role: FOLLOWER
I20260929 19:05:22.000471 22334 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c2402782da146edbcfff12e7037cf1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70fe724446634be9b867efe7c51c76f4" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35161 } } peers { permanent_uuid: "0712bc27ee6340ad85fab7ab0927cc3c" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 36407 } } peers { permanent_uuid: "3c2402782da146edbcfff12e7037cf1d" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46399 } }
I20260929 19:05:22.000813 22334 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c2402782da146edbcfff12e7037cf1d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:05:22.000772 22340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c2402782da146edbcfff12e7037cf1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70fe724446634be9b867efe7c51c76f4" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35161 } } peers { permanent_uuid: "0712bc27ee6340ad85fab7ab0927cc3c" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 36407 } } peers { permanent_uuid: "3c2402782da146edbcfff12e7037cf1d" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46399 } } }
I20260929 19:05:22.000943 22340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c2402782da146edbcfff12e7037cf1d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:05:22.001253 22263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:21.993561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42050 (local address 127.8.158.125:36407)
0929 19:05:21.993596 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:21.993598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:21.993605 (+ 7us) server_negotiation.cc:407] Connection header received
0929 19:05:21.993625 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:21.993626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:21.993660 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:21.993716 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:21.994392 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:21.995065 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:21.995895 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:21.996095 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:21.996168 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:21.998516 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:21.998517 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:21.998517 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:21.998524 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:21.999819 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:22.000197 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:22.000198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:22.000198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:22.000251 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:22.000492 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:22.000493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:22.000493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:22.000650 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:22.000845 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:22.001054 (+ 209us) server_negotiation.cc:299] Negotiation successful
0929 19:05:22.001135 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260929 19:05:22.002740 22351 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3c2402782da146edbcfff12e7037cf1d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 19:05:22.002841 22351 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:05:22.003221 22266 master_runner.cc:186] Error getting master registration for 127.8.158.126:35161: OK, instance_id { permanent_uuid: "70fe724446634be9b867efe7c51c76f4" instance_seqno: 1790708721668245 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 19:05:22.005887 22266 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 19:05:22.007167 8825 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:05:22.011930 22184 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46399 } attempt: 2
I20260929 19:05:22.017225 22338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:22.008616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57950 (local address 127.8.158.124:46399)
0929 19:05:22.008653 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:22.008655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:22.008789 (+ 134us) server_negotiation.cc:407] Connection header received
0929 19:05:22.008845 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:22.008846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:22.008855 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:22.008908 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:22.009939 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:22.010705 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:22.011680 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:22.011955 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:22.012011 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:22.013448 (+ 1437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:22.013449 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:22.013450 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:22.013455 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:22.015728 (+ 2273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:22.016108 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:22.016110 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:22.016110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:22.016169 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:22.016399 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:22.016399 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:22.016400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:22.016558 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:22.016649 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:22.017035 (+ 386us) server_negotiation.cc:299] Negotiation successful
0929 19:05:22.017108 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 19:05:22.018173 22187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:22.009415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56962 (local address 127.8.158.126:35161)
0929 19:05:22.009475 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:22.009477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:22.009878 (+ 401us) server_negotiation.cc:407] Connection header received
0929 19:05:22.009921 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:22.009922 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:22.009932 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:22.009980 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:22.010675 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:22.011353 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:22.012982 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:22.013265 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:22.013307 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:22.014919 (+ 1612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:22.014920 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:22.014920 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:22.014927 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:22.016530 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:22.016922 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:22.016923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:22.016924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:22.016973 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:22.017167 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:22.017167 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:22.017168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:22.017280 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:22.017341 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:22.018011 (+ 670us) server_negotiation.cc:299] Negotiation successful
0929 19:05:22.018084 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 19:05:22.019120 22263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:22.009626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42062 (local address 127.8.158.125:36407)
0929 19:05:22.009692 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:22.009694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:22.010022 (+ 328us) server_negotiation.cc:407] Connection header received
0929 19:05:22.010105 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:22.010106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:22.010115 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:22.010142 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:22.010812 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:22.011488 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:22.012716 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:22.013064 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:22.013108 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:22.015934 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:22.015935 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:22.015936 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:22.015943 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:22.018004 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:22.018302 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:22.018303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:22.018303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:22.018375 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:22.018574 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:22.018575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:22.018575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:22.018684 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:22.018756 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:22.018953 (+ 197us) server_negotiation.cc:299] Negotiation successful
0929 19:05:22.019031 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Sep 29 19:05:22 dist-test-slave-nmsz krb5kdc[22103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708721, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for kudu/127.8.158.124@KRBTEST.COM
I20260929 19:05:22.024227 22186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:22.012160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.124:46399 (local address 127.0.0.1:57956)
0929 19:05:22.012278 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:22.012289 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:22.012327 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:22.013801 (+ 1474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:22.013802 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:22.013953 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:05:22.013989 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:22.013990 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:22.019036 (+ 5046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:22.019037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:22.019623 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:22.019625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:22.019973 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:22.019974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:22.020025 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:22.020417 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:05:22.020421 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:05:22.021620 (+ 1199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:05:22.023242 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:22.023243 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:22.023244 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:22.023410 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:22.023627 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:22.023628 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:22.023628 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:22.023682 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:22.023927 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:05:22.023928 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:05:22.023965 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 19:05:22.024049 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 19:05:22.024135 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260929 19:05:22.024255 22360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:22.012422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57956 (local address 127.8.158.124:46399)
0929 19:05:22.013640 (+ 1218us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:22.013642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:22.013649 (+ 7us) server_negotiation.cc:407] Connection header received
0929 19:05:22.013668 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:22.013668 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:22.013677 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:22.015205 (+ 1528us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:22.015235 (+ 30us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:22.018947 (+ 3712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:22.019700 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:22.019898 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:22.019973 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:22.021718 (+ 1745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:22.021719 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:22.021719 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:22.021726 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:22.023144 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:22.023492 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:22.023493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:22.023494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:22.023552 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:22.023749 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:22.023750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:22.023751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:22.023852 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:22.023922 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:22.024072 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 19:05:22.024140 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1161,"thread_start_us":73,"threads_started":1}
I20260929 19:05:22.025365 22184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70fe724446634be9b867efe7c51c76f4: Bootstrap starting.
I20260929 19:05:22.025933 22184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70fe724446634be9b867efe7c51c76f4: Neither blocks nor log segments found. Creating new log.
I20260929 19:05:22.026198 22184 log.cc:824] T 00000000000000000000000000000000 P 70fe724446634be9b867efe7c51c76f4: Log is configured to *not* fsync() on all Append() calls
I20260929 19:05:22.026895 22184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70fe724446634be9b867efe7c51c76f4: No bootstrap required, opened a new log
I20260929 19:05:22.028522 22184 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70fe724446634be9b867efe7c51c76f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70fe724446634be9b867efe7c51c76f4" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35161 } } peers { permanent_uuid: "0712bc27ee6340ad85fab7ab0927cc3c" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 36407 } } peers { permanent_uuid: "3c2402782da146edbcfff12e7037cf1d" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46399 } }
I20260929 19:05:22.028653 22184 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70fe724446634be9b867efe7c51c76f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:05:22.028712 22184 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70fe724446634be9b867efe7c51c76f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70fe724446634be9b867efe7c51c76f4, State: Initialized, Role: FOLLOWER
I20260929 19:05:22.028854 22184 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70fe724446634be9b867efe7c51c76f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70fe724446634be9b867efe7c51c76f4" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35161 } } peers { permanent_uuid: "0712bc27ee6340ad85fab7ab0927cc3c" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 36407 } } peers { permanent_uuid: "3c2402782da146edbcfff12e7037cf1d" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46399 } }
I20260929 19:05:22.029101 22362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70fe724446634be9b867efe7c51c76f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70fe724446634be9b867efe7c51c76f4" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35161 } } peers { permanent_uuid: "0712bc27ee6340ad85fab7ab0927cc3c" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 36407 } } peers { permanent_uuid: "3c2402782da146edbcfff12e7037cf1d" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46399 } } }
I20260929 19:05:22.029148 22184 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70fe724446634be9b867efe7c51c76f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:05:22.029178 22362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70fe724446634be9b867efe7c51c76f4 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 19:05:22.030872 22373 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 70fe724446634be9b867efe7c51c76f4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 19:05:22.030972 22373 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:05:22.031442 22116 master_runner.cc:186] Error getting master registration for 127.8.158.125:36407: OK, instance_id { permanent_uuid: "0712bc27ee6340ad85fab7ab0927cc3c" instance_seqno: 1790708721808598 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 19:05:22.034065 22116 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 19:05:22.042667 22259 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46399 } attempt: 3
Sep 29 19:05:22 dist-test-slave-nmsz krb5kdc[22103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708721, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.125@KRBTEST.COM for kudu/127.8.158.124@KRBTEST.COM
I20260929 19:05:22.050859 22261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:22.042895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.124:46399 (local address 127.0.0.1:57966)
0929 19:05:22.042937 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:22.042947 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:22.042980 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:22.043133 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:22.043134 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:22.043295 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:05:22.043337 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:22.043339 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:22.044238 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:22.044239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:22.045030 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:22.045032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:22.045530 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:22.045531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:22.045590 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:22.046045 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:05:22.046050 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:05:22.047495 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:05:22.049747 (+ 2252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:22.049748 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:22.049748 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:22.049972 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:22.050180 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:22.050181 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:22.050182 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:22.050247 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:22.050549 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:05:22.050550 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:05:22.050594 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 19:05:22.050668 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 19:05:22.050755 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260929 19:05:22.050873 22360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:22.042972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57966 (local address 127.8.158.124:46399)
0929 19:05:22.043007 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:22.043009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:22.043017 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:05:22.043035 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:22.043036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:22.043057 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:22.043116 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:22.043414 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:22.044156 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:22.045119 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:22.045451 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:22.045521 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:22.047579 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:22.047581 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:22.047581 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:22.047588 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:22.049663 (+ 2075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:22.050049 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:22.050050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:22.050050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:22.050108 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:22.050338 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:22.050339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:22.050339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:22.050476 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:22.050546 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:22.050707 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 19:05:22.050783 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 19:05:22.052501 22259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0712bc27ee6340ad85fab7ab0927cc3c: Bootstrap starting.
I20260929 19:05:22.053331 22259 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0712bc27ee6340ad85fab7ab0927cc3c: Neither blocks nor log segments found. Creating new log.
I20260929 19:05:22.053625 22259 log.cc:824] T 00000000000000000000000000000000 P 0712bc27ee6340ad85fab7ab0927cc3c: Log is configured to *not* fsync() on all Append() calls
I20260929 19:05:22.054503 22259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0712bc27ee6340ad85fab7ab0927cc3c: No bootstrap required, opened a new log
I20260929 19:05:22.056856 22259 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0712bc27ee6340ad85fab7ab0927cc3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70fe724446634be9b867efe7c51c76f4" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35161 } } peers { permanent_uuid: "0712bc27ee6340ad85fab7ab0927cc3c" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 36407 } } peers { permanent_uuid: "3c2402782da146edbcfff12e7037cf1d" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46399 } }
I20260929 19:05:22.056986 22259 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0712bc27ee6340ad85fab7ab0927cc3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:05:22.057020 22259 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0712bc27ee6340ad85fab7ab0927cc3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0712bc27ee6340ad85fab7ab0927cc3c, State: Initialized, Role: FOLLOWER
I20260929 19:05:22.057135 22259 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0712bc27ee6340ad85fab7ab0927cc3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70fe724446634be9b867efe7c51c76f4" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35161 } } peers { permanent_uuid: "0712bc27ee6340ad85fab7ab0927cc3c" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 36407 } } peers { permanent_uuid: "3c2402782da146edbcfff12e7037cf1d" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46399 } }
I20260929 19:05:22.057387 22376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0712bc27ee6340ad85fab7ab0927cc3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70fe724446634be9b867efe7c51c76f4" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35161 } } peers { permanent_uuid: "0712bc27ee6340ad85fab7ab0927cc3c" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 36407 } } peers { permanent_uuid: "3c2402782da146edbcfff12e7037cf1d" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46399 } } }
I20260929 19:05:22.057449 22259 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0712bc27ee6340ad85fab7ab0927cc3c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:05:22.058378 22376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0712bc27ee6340ad85fab7ab0927cc3c [sys.catalog]: This master's current role is: FOLLOWER
W20260929 19:05:22.059367 22387 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0712bc27ee6340ad85fab7ab0927cc3c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 19:05:22.059417 22387 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:05:22.061554 22191 master_runner.cc:263] No leader master found from master 0712bc27ee6340ad85fab7ab0927cc3c
I20260929 19:05:22.061601 22191 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 19:05:22.068763 22340 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c2402782da146edbcfff12e7037cf1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:05:22.068852 22340 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c2402782da146edbcfff12e7037cf1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70fe724446634be9b867efe7c51c76f4" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35161 } } peers { permanent_uuid: "0712bc27ee6340ad85fab7ab0927cc3c" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 36407 } } peers { permanent_uuid: "3c2402782da146edbcfff12e7037cf1d" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46399 } }
I20260929 19:05:22.069033 22340 leader_election.cc:288] T 00000000000000000000000000000000 P 3c2402782da146edbcfff12e7037cf1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70fe724446634be9b867efe7c51c76f4 (127.8.158.126:35161), 0712bc27ee6340ad85fab7ab0927cc3c (127.8.158.125:36407)
I20260929 19:05:22.069442 22158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3c2402782da146edbcfff12e7037cf1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70fe724446634be9b867efe7c51c76f4" is_pre_election: true
I20260929 19:05:22.069564 22158 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 70fe724446634be9b867efe7c51c76f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c2402782da146edbcfff12e7037cf1d in term 0.
I20260929 19:05:22.069729 22233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3c2402782da146edbcfff12e7037cf1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0712bc27ee6340ad85fab7ab0927cc3c" is_pre_election: true
I20260929 19:05:22.069831 22233 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0712bc27ee6340ad85fab7ab0927cc3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c2402782da146edbcfff12e7037cf1d in term 0.
I20260929 19:05:22.070238 22285 leader_election.cc:302] T 00000000000000000000000000000000 P 3c2402782da146edbcfff12e7037cf1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c2402782da146edbcfff12e7037cf1d, 70fe724446634be9b867efe7c51c76f4; no voters:
I20260929 19:05:22.070464 22340 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c2402782da146edbcfff12e7037cf1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:05:22.070515 22340 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c2402782da146edbcfff12e7037cf1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:05:22.070537 22340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c2402782da146edbcfff12e7037cf1d [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:05:22.071260 22340 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c2402782da146edbcfff12e7037cf1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70fe724446634be9b867efe7c51c76f4" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35161 } } peers { permanent_uuid: "0712bc27ee6340ad85fab7ab0927cc3c" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 36407 } } peers { permanent_uuid: "3c2402782da146edbcfff12e7037cf1d" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46399 } }
I20260929 19:05:22.071362 22340 leader_election.cc:288] T 00000000000000000000000000000000 P 3c2402782da146edbcfff12e7037cf1d [CANDIDATE]: Term 1 election: Requested vote from peers 70fe724446634be9b867efe7c51c76f4 (127.8.158.126:35161), 0712bc27ee6340ad85fab7ab0927cc3c (127.8.158.125:36407)
I20260929 19:05:22.071682 22158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3c2402782da146edbcfff12e7037cf1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70fe724446634be9b867efe7c51c76f4"
I20260929 19:05:22.071761 22158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70fe724446634be9b867efe7c51c76f4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:05:22.072559 22158 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 70fe724446634be9b867efe7c51c76f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c2402782da146edbcfff12e7037cf1d in term 1.
I20260929 19:05:22.072727 22233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3c2402782da146edbcfff12e7037cf1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0712bc27ee6340ad85fab7ab0927cc3c"
I20260929 19:05:22.072789 22233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0712bc27ee6340ad85fab7ab0927cc3c [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:05:22.073575 22233 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0712bc27ee6340ad85fab7ab0927cc3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c2402782da146edbcfff12e7037cf1d in term 1.
I20260929 19:05:22.073792 22285 leader_election.cc:302] T 00000000000000000000000000000000 P 3c2402782da146edbcfff12e7037cf1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0712bc27ee6340ad85fab7ab0927cc3c, 3c2402782da146edbcfff12e7037cf1d; no voters:
I20260929 19:05:22.073920 22340 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c2402782da146edbcfff12e7037cf1d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:05:22.074041 22340 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c2402782da146edbcfff12e7037cf1d [term 1 LEADER]: Becoming Leader. State: Replica: 3c2402782da146edbcfff12e7037cf1d, State: Running, Role: LEADER
I20260929 19:05:22.074112 22340 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c2402782da146edbcfff12e7037cf1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70fe724446634be9b867efe7c51c76f4" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35161 } } peers { permanent_uuid: "0712bc27ee6340ad85fab7ab0927cc3c" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 36407 } } peers { permanent_uuid: "3c2402782da146edbcfff12e7037cf1d" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46399 } }
I20260929 19:05:22.074579 22340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c2402782da146edbcfff12e7037cf1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c2402782da146edbcfff12e7037cf1d. Latest consensus state: current_term: 1 leader_uuid: "3c2402782da146edbcfff12e7037cf1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70fe724446634be9b867efe7c51c76f4" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35161 } } peers { permanent_uuid: "0712bc27ee6340ad85fab7ab0927cc3c" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 36407 } } peers { permanent_uuid: "3c2402782da146edbcfff12e7037cf1d" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46399 } } }
I20260929 19:05:22.074667 22340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c2402782da146edbcfff12e7037cf1d [sys.catalog]: This master's current role is: LEADER
I20260929 19:05:22.074905 22391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:05:22.075088 22391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:05:22.077548 22233 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0712bc27ee6340ad85fab7ab0927cc3c [term 1 FOLLOWER]: Refusing update from remote peer 3c2402782da146edbcfff12e7037cf1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:05:22.077863 22340 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3c2402782da146edbcfff12e7037cf1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0712bc27ee6340ad85fab7ab0927cc3c" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 36407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:05:22.079006 22376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0712bc27ee6340ad85fab7ab0927cc3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c2402782da146edbcfff12e7037cf1d. Latest consensus state: current_term: 1 leader_uuid: "3c2402782da146edbcfff12e7037cf1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70fe724446634be9b867efe7c51c76f4" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35161 } } peers { permanent_uuid: "0712bc27ee6340ad85fab7ab0927cc3c" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 36407 } } peers { permanent_uuid: "3c2402782da146edbcfff12e7037cf1d" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46399 } } }
I20260929 19:05:22.079082 22376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0712bc27ee6340ad85fab7ab0927cc3c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:05:22.079488 22158 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 70fe724446634be9b867efe7c51c76f4 [term 1 FOLLOWER]: Refusing update from remote peer 3c2402782da146edbcfff12e7037cf1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:05:22.079754 22340 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3c2402782da146edbcfff12e7037cf1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "70fe724446634be9b867efe7c51c76f4" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:05:22.080914 22362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70fe724446634be9b867efe7c51c76f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c2402782da146edbcfff12e7037cf1d. Latest consensus state: current_term: 1 leader_uuid: "3c2402782da146edbcfff12e7037cf1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70fe724446634be9b867efe7c51c76f4" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35161 } } peers { permanent_uuid: "0712bc27ee6340ad85fab7ab0927cc3c" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 36407 } } peers { permanent_uuid: "3c2402782da146edbcfff12e7037cf1d" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46399 } } }
I20260929 19:05:22.081007 22362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70fe724446634be9b867efe7c51c76f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:05:22.082096 22392 mvcc.cc:205] Tried to move back new op lower bound from 7334742925627768832 to 7334742925616152576. Current Snapshot: MvccSnapshot[applied={T|T < 7334742925627768832}]
I20260929 19:05:22.082450 22340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c2402782da146edbcfff12e7037cf1d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3c2402782da146edbcfff12e7037cf1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70fe724446634be9b867efe7c51c76f4" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35161 } } peers { permanent_uuid: "0712bc27ee6340ad85fab7ab0927cc3c" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 36407 } } peers { permanent_uuid: "3c2402782da146edbcfff12e7037cf1d" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46399 } } }
I20260929 19:05:22.082522 22340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c2402782da146edbcfff12e7037cf1d [sys.catalog]: This master's current role is: LEADER
I20260929 19:05:22.082935 22376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0712bc27ee6340ad85fab7ab0927cc3c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3c2402782da146edbcfff12e7037cf1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70fe724446634be9b867efe7c51c76f4" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35161 } } peers { permanent_uuid: "0712bc27ee6340ad85fab7ab0927cc3c" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 36407 } } peers { permanent_uuid: "3c2402782da146edbcfff12e7037cf1d" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46399 } } }
I20260929 19:05:22.083021 22376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0712bc27ee6340ad85fab7ab0927cc3c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:05:22.083596 22393 mvcc.cc:205] Tried to move back new op lower bound from 7334742925627768832 to 7334742925616152576. Current Snapshot: MvccSnapshot[applied={T|T < 7334742925627768832}]
I20260929 19:05:22.084436 22340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c2402782da146edbcfff12e7037cf1d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3c2402782da146edbcfff12e7037cf1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70fe724446634be9b867efe7c51c76f4" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35161 } } peers { permanent_uuid: "0712bc27ee6340ad85fab7ab0927cc3c" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 36407 } } peers { permanent_uuid: "3c2402782da146edbcfff12e7037cf1d" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46399 } } }
I20260929 19:05:22.084511 22340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c2402782da146edbcfff12e7037cf1d [sys.catalog]: This master's current role is: LEADER
I20260929 19:05:22.084924 22362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70fe724446634be9b867efe7c51c76f4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3c2402782da146edbcfff12e7037cf1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70fe724446634be9b867efe7c51c76f4" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35161 } } peers { permanent_uuid: "0712bc27ee6340ad85fab7ab0927cc3c" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 36407 } } peers { permanent_uuid: "3c2402782da146edbcfff12e7037cf1d" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 46399 } } }
I20260929 19:05:22.084995 22362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70fe724446634be9b867efe7c51c76f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:05:22.085284 22395 mvcc.cc:205] Tried to move back new op lower bound from 7334742925627768832 to 7334742925616152576. Current Snapshot: MvccSnapshot[applied={T|T < 7334742925627768832}]
I20260929 19:05:22.086095 22391 catalog_manager.cc:1347] Generated new cluster ID: 15f288d0f93f4d15b7f786081209d980
I20260929 19:05:22.086159 22391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:05:22.116088 22391 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:05:22.116859 22391 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:05:22.125663 22391 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c2402782da146edbcfff12e7037cf1d: Generated new TSK 0
I20260929 19:05:22.125900 22391 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:05:23.031453 22373 catalog_manager.cc:1259] Loaded cluster ID: 15f288d0f93f4d15b7f786081209d980
I20260929 19:05:23.031533 22373 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 70fe724446634be9b867efe7c51c76f4: loading cluster ID for follower catalog manager: success
I20260929 19:05:23.032357 22373 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 70fe724446634be9b867efe7c51c76f4: acquiring CA information for follower catalog manager: success
I20260929 19:05:23.032809 22373 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 70fe724446634be9b867efe7c51c76f4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 19:05:23.059813 22387 catalog_manager.cc:1259] Loaded cluster ID: 15f288d0f93f4d15b7f786081209d980
I20260929 19:05:23.059908 22387 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0712bc27ee6340ad85fab7ab0927cc3c: loading cluster ID for follower catalog manager: success
I20260929 19:05:23.060724 22387 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0712bc27ee6340ad85fab7ab0927cc3c: acquiring CA information for follower catalog manager: success
I20260929 19:05:23.061028 22387 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0712bc27ee6340ad85fab7ab0927cc3c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 19:05:23.183876 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 22116
I20260929 19:05:23.188937 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 22191
I20260929 19:05:23.193524 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 22266
2026-09-29T19:05:23Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2810 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708573736807-8825-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:05:23 dist-test-slave-nmsz krb5kdc[22407](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:05:23 dist-test-slave-nmsz krb5kdc[22407](info): set up 2 sockets
Sep 29 19:05:23 dist-test-slave-nmsz krb5kdc[22407](info): commencing operation
krb5kdc: starting...
W20260929 19:05:24.238399 8825 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Sep 29 19:05:24 dist-test-slave-nmsz krb5kdc[22407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:05:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:05:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.158.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
I20260929 19:05:24.350975 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:41231
--webserver_interface=127.8.158.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:46167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:41231
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:05:24.430154 22420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:05:24.430378 22420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:05:24.430401 22420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:05:24.431461 22420 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:05:24.431495 22420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:05:24.431509 22420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:05:24.431522 22420 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:05:24.431535 22420 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:05:24.432869 22420 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:46167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:41231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:41231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.158.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:05:24.433043 22420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:05:24.433204 22420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:05:24.437968 22426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:24.438030 22428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:24.437983 22425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:05:24.438535 22420 server_base.cc:1034] running on GCE node
I20260929 19:05:24.438813 22420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:05:24.439116 22420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:05:24.440382 22420 hybrid_clock.cc:648] HybridClock initialized: now 1790708724440362 us; error 39 us; skew 500 ppm
Sep 29 19:05:24 dist-test-slave-nmsz krb5kdc[22407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708724, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:05:24.442983 22420 init.cc:377] Logged in from keytab as kudu/127.8.158.126@KRBTEST.COM (short username kudu)
I20260929 19:05:24.443451 22420 webserver.cc:460] Webserver started at http://127.8.158.126:33289/ using document root <none> and password file <none>
I20260929 19:05:24.443663 22420 fs_manager.cc:359] Metadata directory not provided
I20260929 19:05:24.443709 22420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:05:24.443857 22420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:05:24.444877 22420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "f46abe0e33ec4f6bb767297d73d5d0e1"
format_stamp: "Formatted at 2026-09-29 19:05:24 on dist-test-slave-nmsz"
server_key: "61a80b54796fc5f0b05cf514de25cd8b"
server_key_iv: "27f91b96152e771064b079ceb55b2c4f"
server_key_version: "encryptionkey@0"
I20260929 19:05:24.445241 22420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "f46abe0e33ec4f6bb767297d73d5d0e1"
format_stamp: "Formatted at 2026-09-29 19:05:24 on dist-test-slave-nmsz"
server_key: "61a80b54796fc5f0b05cf514de25cd8b"
server_key_iv: "27f91b96152e771064b079ceb55b2c4f"
server_key_version: "encryptionkey@0"
I20260929 19:05:24.446602 22420 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:05:24.447386 22435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:24.447575 22420 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:05:24.447674 22420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "f46abe0e33ec4f6bb767297d73d5d0e1"
format_stamp: "Formatted at 2026-09-29 19:05:24 on dist-test-slave-nmsz"
server_key: "61a80b54796fc5f0b05cf514de25cd8b"
server_key_iv: "27f91b96152e771064b079ceb55b2c4f"
server_key_version: "encryptionkey@0"
I20260929 19:05:24.447750 22420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:05:24.470872 22420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:05:24.472831 22420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:05:24.472988 22420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:05:24.476943 22420 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.126:41231
I20260929 19:05:24.476974 22487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.126:41231 every 8 connection(s)
I20260929 19:05:24.477305 22420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:05:24.478091 22488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:05:24.480281 22488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f46abe0e33ec4f6bb767297d73d5d0e1: Bootstrap starting.
I20260929 19:05:24.480835 22488 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f46abe0e33ec4f6bb767297d73d5d0e1: Neither blocks nor log segments found. Creating new log.
I20260929 19:05:24.481110 22488 log.cc:824] T 00000000000000000000000000000000 P f46abe0e33ec4f6bb767297d73d5d0e1: Log is configured to *not* fsync() on all Append() calls
I20260929 19:05:24.481781 22488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f46abe0e33ec4f6bb767297d73d5d0e1: No bootstrap required, opened a new log
I20260929 19:05:24.483392 22488 raft_consensus.cc:348] T 00000000000000000000000000000000 P f46abe0e33ec4f6bb767297d73d5d0e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46abe0e33ec4f6bb767297d73d5d0e1" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 41231 } }
I20260929 19:05:24.483518 22488 raft_consensus.cc:374] T 00000000000000000000000000000000 P f46abe0e33ec4f6bb767297d73d5d0e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:05:24.483541 22488 raft_consensus.cc:729] T 00000000000000000000000000000000 P f46abe0e33ec4f6bb767297d73d5d0e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46abe0e33ec4f6bb767297d73d5d0e1, State: Initialized, Role: FOLLOWER
I20260929 19:05:24.483706 22488 consensus_queue.cc:260] T 00000000000000000000000000000000 P f46abe0e33ec4f6bb767297d73d5d0e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46abe0e33ec4f6bb767297d73d5d0e1" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 41231 } }
I20260929 19:05:24.483783 22488 raft_consensus.cc:388] T 00000000000000000000000000000000 P f46abe0e33ec4f6bb767297d73d5d0e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:05:24.483826 22488 raft_consensus.cc:482] T 00000000000000000000000000000000 P f46abe0e33ec4f6bb767297d73d5d0e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:05:24.483875 22488 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f46abe0e33ec4f6bb767297d73d5d0e1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:05:24.484371 22488 raft_consensus.cc:504] T 00000000000000000000000000000000 P f46abe0e33ec4f6bb767297d73d5d0e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46abe0e33ec4f6bb767297d73d5d0e1" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 41231 } }
I20260929 19:05:24.484475 22488 leader_election.cc:302] T 00000000000000000000000000000000 P f46abe0e33ec4f6bb767297d73d5d0e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f46abe0e33ec4f6bb767297d73d5d0e1; no voters:
I20260929 19:05:24.484639 22488 leader_election.cc:288] T 00000000000000000000000000000000 P f46abe0e33ec4f6bb767297d73d5d0e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:05:24.484668 22491 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f46abe0e33ec4f6bb767297d73d5d0e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:05:24.484862 22491 raft_consensus.cc:686] T 00000000000000000000000000000000 P f46abe0e33ec4f6bb767297d73d5d0e1 [term 1 LEADER]: Becoming Leader. State: Replica: f46abe0e33ec4f6bb767297d73d5d0e1, State: Running, Role: LEADER
I20260929 19:05:24.484894 22488 sys_catalog.cc:564] T 00000000000000000000000000000000 P f46abe0e33ec4f6bb767297d73d5d0e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:05:24.484947 22491 consensus_queue.cc:237] T 00000000000000000000000000000000 P f46abe0e33ec4f6bb767297d73d5d0e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46abe0e33ec4f6bb767297d73d5d0e1" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 41231 } }
I20260929 19:05:24.485097 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 22420
I20260929 19:05:24.485198 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:05:24.485381 8825 external_mini_cluster.cc:1428] Setting key 4b82217e5345efda9a76df3ef40fe7a1
I20260929 19:05:24.486084 22493 sys_catalog.cc:455] T 00000000000000000000000000000000 P f46abe0e33ec4f6bb767297d73d5d0e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f46abe0e33ec4f6bb767297d73d5d0e1. Latest consensus state: current_term: 1 leader_uuid: "f46abe0e33ec4f6bb767297d73d5d0e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46abe0e33ec4f6bb767297d73d5d0e1" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 41231 } } }
I20260929 19:05:24.486232 22493 sys_catalog.cc:458] T 00000000000000000000000000000000 P f46abe0e33ec4f6bb767297d73d5d0e1 [sys.catalog]: This master's current role is: LEADER
I20260929 19:05:24.486121 22492 sys_catalog.cc:455] T 00000000000000000000000000000000 P f46abe0e33ec4f6bb767297d73d5d0e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f46abe0e33ec4f6bb767297d73d5d0e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46abe0e33ec4f6bb767297d73d5d0e1" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 41231 } } }
I20260929 19:05:24.486615 22492 sys_catalog.cc:458] T 00000000000000000000000000000000 P f46abe0e33ec4f6bb767297d73d5d0e1 [sys.catalog]: This master's current role is: LEADER
I20260929 19:05:24.486577 22509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:05:24.487144 22509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:05:24.488629 22509 catalog_manager.cc:1347] Generated new cluster ID: a63b2a023e7c4310b8121d073547af43
I20260929 19:05:24.488680 22509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 19:05:24 dist-test-slave-nmsz krb5kdc[22407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:05:24.495684 22502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:24.485814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39666 (local address 127.8.158.126:41231)
0929 19:05:24.486174 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:24.486183 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:24.486213 (+ 30us) server_negotiation.cc:407] Connection header received
0929 19:05:24.486353 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:24.486360 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:24.486439 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:24.486616 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:24.487351 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:24.488569 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:24.489319 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:24.489738 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:24.489854 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:24.492420 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:24.492423 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:24.492423 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:24.492433 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:24.494075 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:24.494514 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:24.494517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:24.494518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:24.494617 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:24.494845 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:24.494845 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:24.494846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:24.495070 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:24.495239 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:24.495341 (+ 102us) server_negotiation.cc:299] Negotiation successful
0929 19:05:24.495479 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":311,"thread_start_us":100,"threads_started":1}
I20260929 19:05:24.496613 8825 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:05:24.504086 22509 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:05:24.504585 22509 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:05:24.509999 22509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f46abe0e33ec4f6bb767297d73d5d0e1: Generated new TSK 0
I20260929 19:05:24.510174 22509 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:05:24.550364 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 22420
2026-09-29T19:05:24Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1355 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:05:24 dist-test-slave-nmsz krb5kdc[22515](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:05:24 dist-test-slave-nmsz krb5kdc[22515](info): set up 2 sockets
Sep 29 19:05:24 dist-test-slave-nmsz krb5kdc[22515](info): commencing operation
krb5kdc: starting...
W20260929 19:05:25.578382 8825 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Sep 29 19:05:25 dist-test-slave-nmsz krb5kdc[22515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:05:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:05:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.158.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
I20260929 19:05:25.697067 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:38373
--webserver_interface=127.8.158.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:34333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:38373
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:05:25.776659 22528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:05:25.776867 22528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:05:25.776904 22528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:05:25.778023 22528 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:05:25.778084 22528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:05:25.778110 22528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:05:25.778198 22528 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:05:25.778240 22528 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:05:25.779706 22528 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:34333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:38373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:38373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.158.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:05:25.779958 22528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:05:25.780182 22528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:05:25.785041 22536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:25.785036 22534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:25.785023 22533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:05:25.785257 22528 server_base.cc:1034] running on GCE node
I20260929 19:05:25.785509 22528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:05:25.785734 22528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:05:25.786892 22528 hybrid_clock.cc:648] HybridClock initialized: now 1790708725786880 us; error 27 us; skew 500 ppm
Sep 29 19:05:25 dist-test-slave-nmsz krb5kdc[22515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708725, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:05:25.793215 22528 init.cc:377] Logged in from keytab as kudu/127.8.158.126@KRBTEST.COM (short username kudu)
I20260929 19:05:25.793665 22528 webserver.cc:460] Webserver started at http://127.8.158.126:33551/ using document root <none> and password file <none>
I20260929 19:05:25.793890 22528 fs_manager.cc:359] Metadata directory not provided
I20260929 19:05:25.793936 22528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:05:25.794094 22528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:05:25.795125 22528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "7181e789b0c24f22b65f8fdd3e231054"
format_stamp: "Formatted at 2026-09-29 19:05:25 on dist-test-slave-nmsz"
server_key: "327656b0ce57b1499e5606bdaeaa9ab5"
server_key_iv: "225f646728bcf3f8368a57734be38341"
server_key_version: "encryptionkey@0"
I20260929 19:05:25.795467 22528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "7181e789b0c24f22b65f8fdd3e231054"
format_stamp: "Formatted at 2026-09-29 19:05:25 on dist-test-slave-nmsz"
server_key: "327656b0ce57b1499e5606bdaeaa9ab5"
server_key_iv: "225f646728bcf3f8368a57734be38341"
server_key_version: "encryptionkey@0"
I20260929 19:05:25.796766 22528 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:05:25.797587 22543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:25.797803 22528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:05:25.797910 22528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "7181e789b0c24f22b65f8fdd3e231054"
format_stamp: "Formatted at 2026-09-29 19:05:25 on dist-test-slave-nmsz"
server_key: "327656b0ce57b1499e5606bdaeaa9ab5"
server_key_iv: "225f646728bcf3f8368a57734be38341"
server_key_version: "encryptionkey@0"
I20260929 19:05:25.797989 22528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:05:25.808846 22528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:05:25.810770 22528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:05:25.810917 22528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:05:25.814781 22528 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.126:38373
I20260929 19:05:25.814852 22595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.126:38373 every 8 connection(s)
I20260929 19:05:25.815129 22528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:05:25.816028 22596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:05:25.818050 22596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054: Bootstrap starting.
I20260929 19:05:25.818602 22596 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054: Neither blocks nor log segments found. Creating new log.
I20260929 19:05:25.818830 22596 log.cc:824] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054: Log is configured to *not* fsync() on all Append() calls
I20260929 19:05:25.819424 22596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054: No bootstrap required, opened a new log
I20260929 19:05:25.820883 22596 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7181e789b0c24f22b65f8fdd3e231054" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 38373 } }
I20260929 19:05:25.821005 22596 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:05:25.821026 22596 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7181e789b0c24f22b65f8fdd3e231054, State: Initialized, Role: FOLLOWER
I20260929 19:05:25.821151 22596 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7181e789b0c24f22b65f8fdd3e231054" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 38373 } }
I20260929 19:05:25.821234 22596 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:05:25.821257 22596 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:05:25.821326 22596 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:05:25.821355 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 22528
I20260929 19:05:25.821462 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:05:25.821638 8825 external_mini_cluster.cc:1428] Setting key 185c7c9ae47d9b63b47c2c978480b09f
I20260929 19:05:25.822405 22596 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7181e789b0c24f22b65f8fdd3e231054" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 38373 } }
I20260929 19:05:25.822507 22596 leader_election.cc:302] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7181e789b0c24f22b65f8fdd3e231054; no voters:
I20260929 19:05:25.822654 22596 leader_election.cc:288] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:05:25.822701 22601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:05:25.822842 22601 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [term 1 LEADER]: Becoming Leader. State: Replica: 7181e789b0c24f22b65f8fdd3e231054, State: Running, Role: LEADER
I20260929 19:05:25.822896 22596 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:05:25.822932 22601 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7181e789b0c24f22b65f8fdd3e231054" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 38373 } }
I20260929 19:05:25.823251 22601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7181e789b0c24f22b65f8fdd3e231054. Latest consensus state: current_term: 1 leader_uuid: "7181e789b0c24f22b65f8fdd3e231054" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7181e789b0c24f22b65f8fdd3e231054" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 38373 } } }
I20260929 19:05:25.823335 22601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [sys.catalog]: This master's current role is: LEADER
I20260929 19:05:25.823258 22602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7181e789b0c24f22b65f8fdd3e231054" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7181e789b0c24f22b65f8fdd3e231054" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 38373 } } }
I20260929 19:05:25.823504 22602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [sys.catalog]: This master's current role is: LEADER
I20260929 19:05:25.823534 22608 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:05:25.824090 22608 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:05:25.825954 22608 catalog_manager.cc:1347] Generated new cluster ID: 1cb79e5cf82d40e78d3c5340b60f0ca3
I20260929 19:05:25.826009 22608 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 19:05:25 dist-test-slave-nmsz krb5kdc[22515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:05:25.831929 22600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:25.822087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57752 (local address 127.8.158.126:38373)
0929 19:05:25.822271 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:25.822280 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:25.822330 (+ 50us) server_negotiation.cc:407] Connection header received
0929 19:05:25.822439 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:25.822446 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:25.822500 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:25.822653 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:25.823512 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:25.824866 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:25.825893 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:25.826365 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:25.826504 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:25.828717 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:25.828721 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:25.828721 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:25.828731 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:25.830471 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:25.830841 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:25.830844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:25.830844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:25.830940 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:25.831149 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:25.831150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:25.831150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:25.831373 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:25.831548 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:25.831610 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 19:05:25.831743 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
I20260929 19:05:25.847527 22608 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:05:25.848212 22608 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:05:25.857527 22608 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054: Generated new TSK 0
I20260929 19:05:25.857723 22608 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.158.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab.
Entry for principal kudu/127.8.158.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab.
I20260929 19:05:25.885432 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.65:0
--local_ip_for_outbound_sockets=127.8.158.65
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:38373
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab
--principal=kudu/127.8.158.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:34333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:05:25.965083 22622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:05:25.965310 22622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:05:25.965335 22622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:05:25.966502 22622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:05:25.966542 22622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:05:25.966568 22622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.65
I20260929 19:05:25.967988 22622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:34333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab
--principal=kudu/127.8.158.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:38373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.65
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:05:25.968225 22622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:05:25.968463 22622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:05:25.973351 22627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:25.973328 22630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:05:25.973453 22622 server_base.cc:1034] running on GCE node
W20260929 19:05:25.973307 22628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:05:25.973731 22622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:05:25.973922 22622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:05:25.975049 22622 hybrid_clock.cc:648] HybridClock initialized: now 1790708725975040 us; error 27 us; skew 500 ppm
Sep 29 19:05:25 dist-test-slave-nmsz krb5kdc[22515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708725, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:05:25.977481 22622 init.cc:377] Logged in from keytab as kudu/127.8.158.65@KRBTEST.COM (short username kudu)
I20260929 19:05:25.977871 22622 webserver.cc:460] Webserver started at http://127.8.158.65:43355/ using document root <none> and password file <none>
I20260929 19:05:25.978044 22622 fs_manager.cc:359] Metadata directory not provided
I20260929 19:05:25.978081 22622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:05:25.978163 22622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:05:25.979151 22622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-0/data/instance:
uuid: "ffeec491014e4d6b89512c09a168dc9b"
format_stamp: "Formatted at 2026-09-29 19:05:25 on dist-test-slave-nmsz"
server_key: "c303953b3fd52bb2c7c754497c42d7fa"
server_key_iv: "bc0a2b9477c4cca25bed3ec569d2e990"
server_key_version: "encryptionkey@0"
I20260929 19:05:25.979493 22622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance:
uuid: "ffeec491014e4d6b89512c09a168dc9b"
format_stamp: "Formatted at 2026-09-29 19:05:25 on dist-test-slave-nmsz"
server_key: "c303953b3fd52bb2c7c754497c42d7fa"
server_key_iv: "bc0a2b9477c4cca25bed3ec569d2e990"
server_key_version: "encryptionkey@0"
I20260929 19:05:25.980751 22622 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:05:25.981617 22637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:25.982038 22622 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:05:25.982223 22622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-0/wal
uuid: "ffeec491014e4d6b89512c09a168dc9b"
format_stamp: "Formatted at 2026-09-29 19:05:25 on dist-test-slave-nmsz"
server_key: "c303953b3fd52bb2c7c754497c42d7fa"
server_key_iv: "bc0a2b9477c4cca25bed3ec569d2e990"
server_key_version: "encryptionkey@0"
I20260929 19:05:25.982384 22622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:05:25.989360 22622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:05:25.991319 22622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:05:25.991590 22622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:05:25.991833 22622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:05:25.992159 22622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:05:25.992255 22622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:25.992301 22622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:05:25.992326 22622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:25.998085 22622 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.65:46257
I20260929 19:05:25.998127 22750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.65:46257 every 8 connection(s)
I20260929 19:05:25.998514 22622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
I20260929 19:05:25.999866 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 22622
I20260929 19:05:25.999961 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance
I20260929 19:05:26.000155 8825 external_mini_cluster.cc:1428] Setting key e929bf1115ff0198eded7e635668fdd0
Sep 29 19:05:25 dist-test-slave-nmsz krb5kdc[22515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708725, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:05:26.008697 22600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:25.999185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:50863 (local address 127.8.158.126:38373)
0929 19:05:25.999248 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:25.999251 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:25.999411 (+ 160us) server_negotiation.cc:407] Connection header received
0929 19:05:26.000111 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:26.000113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:26.000133 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:26.000186 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:26.001357 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.001940 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.003126 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.003344 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.003429 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:26.005502 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:26.005504 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:26.005504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:26.005511 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:26.007168 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:26.007585 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:26.007586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:26.007587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:26.007643 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:26.007931 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:26.007931 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:26.007932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:26.008064 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:26.008128 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:26.008488 (+ 360us) server_negotiation.cc:299] Negotiation successful
0929 19:05:26.008574 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 19:05:26.008877 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:25.999124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.65:50863)
0929 19:05:25.999287 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:25.999330 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:25.999966 (+ 636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:26.000265 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:26.000270 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:26.000981 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:05:26.001261 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:26.001268 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.002134 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.002135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:26.003021 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:26.003024 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.003440 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.003441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:26.003561 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:26.004086 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:05:26.004094 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:05:26.005406 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:05:26.007280 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:26.007286 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:26.007288 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:26.007496 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:26.007724 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:26.007725 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:26.007725 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:26.007853 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:26.008156 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:05:26.008163 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:05:26.008326 (+ 163us) client_negotiation.cc:769] Sending connection context
0929 19:05:26.008439 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 19:05:26.008602 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":71,"threads_started":1}
WARNING: no policy specified for kudu/127.8.158.66@KRBTEST.COM; defaulting to no policy
I20260929 19:05:26.009697 22751 heartbeater.cc:347] Connected to a master server at 127.8.158.126:38373
I20260929 19:05:26.009796 22751 heartbeater.cc:464] Registering TS with master...
I20260929 19:05:26.009965 22751 heartbeater.cc:511] Master 127.8.158.126:38373 requested a full tablet report, sending...
I20260929 19:05:26.010491 22560 ts_manager.cc:194] Registered new tserver with Master: ffeec491014e4d6b89512c09a168dc9b (127.8.158.65:46257)
I20260929 19:05:26.011157 22560 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.65@KRBTEST.COM'} at 127.8.158.65:50863
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab.
Entry for principal kudu/127.8.158.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab.
I20260929 19:05:26.027235 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.66:0
--local_ip_for_outbound_sockets=127.8.158.66
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:38373
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab
--principal=kudu/127.8.158.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:34333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:05:26.105589 22756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:05:26.105777 22756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:05:26.105799 22756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:05:26.106899 22756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:05:26.106935 22756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:05:26.106961 22756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.66
I20260929 19:05:26.108305 22756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:34333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab
--principal=kudu/127.8.158.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:38373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.66
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:05:26.108474 22756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:05:26.108630 22756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:05:26.113647 22764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:26.113713 22761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:05:26.113776 22756 server_base.cc:1034] running on GCE node
W20260929 19:05:26.113710 22762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:05:26.114202 22756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:05:26.114496 22756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:05:26.115665 22756 hybrid_clock.cc:648] HybridClock initialized: now 1790708726115661 us; error 37 us; skew 500 ppm
Sep 29 19:05:26 dist-test-slave-nmsz krb5kdc[22515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708726, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:05:26.118038 22756 init.cc:377] Logged in from keytab as kudu/127.8.158.66@KRBTEST.COM (short username kudu)
I20260929 19:05:26.118551 22756 webserver.cc:460] Webserver started at http://127.8.158.66:32849/ using document root <none> and password file <none>
I20260929 19:05:26.118772 22756 fs_manager.cc:359] Metadata directory not provided
I20260929 19:05:26.118815 22756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:05:26.118937 22756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:05:26.119935 22756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-1/data/instance:
uuid: "bbdc71811c3b4416a687f18009ebc913"
format_stamp: "Formatted at 2026-09-29 19:05:26 on dist-test-slave-nmsz"
server_key: "860585501b14d580da7a42459a7852e0"
server_key_iv: "1687d7ca24be53ddd9e1f1c6b371f095"
server_key_version: "encryptionkey@0"
I20260929 19:05:26.120275 22756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance:
uuid: "bbdc71811c3b4416a687f18009ebc913"
format_stamp: "Formatted at 2026-09-29 19:05:26 on dist-test-slave-nmsz"
server_key: "860585501b14d580da7a42459a7852e0"
server_key_iv: "1687d7ca24be53ddd9e1f1c6b371f095"
server_key_version: "encryptionkey@0"
I20260929 19:05:26.121613 22756 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:05:26.122422 22771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:26.122598 22756 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:05:26.122691 22756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-1/wal
uuid: "bbdc71811c3b4416a687f18009ebc913"
format_stamp: "Formatted at 2026-09-29 19:05:26 on dist-test-slave-nmsz"
server_key: "860585501b14d580da7a42459a7852e0"
server_key_iv: "1687d7ca24be53ddd9e1f1c6b371f095"
server_key_version: "encryptionkey@0"
I20260929 19:05:26.122766 22756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:05:26.132174 22756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:05:26.133878 22756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:05:26.133980 22756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:05:26.134142 22756 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:05:26.134490 22756 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:05:26.134526 22756 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:26.134545 22756 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:05:26.134593 22756 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:26.140133 22756 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.66:38085
I20260929 19:05:26.140213 22884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.66:38085 every 8 connection(s)
I20260929 19:05:26.140503 22756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
I20260929 19:05:26.141484 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 22756
I20260929 19:05:26.141601 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance
I20260929 19:05:26.141782 8825 external_mini_cluster.cc:1428] Setting key ac2faf7a313effaaf050686fb05278ca
Sep 29 19:05:26 dist-test-slave-nmsz krb5kdc[22515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708726, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.66@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.158.67@KRBTEST.COM; defaulting to no policy
I20260929 19:05:26.150280 22600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:26.141207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:49535 (local address 127.8.158.126:38373)
0929 19:05:26.141277 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:26.141279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:26.141466 (+ 187us) server_negotiation.cc:407] Connection header received
0929 19:05:26.142080 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:26.142081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:26.142092 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:26.142166 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:26.143180 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.143666 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.144634 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.144834 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.144908 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:26.147445 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:26.147446 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:26.147447 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:26.147453 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:26.148807 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:26.149308 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:26.149309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:26.149309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:26.149373 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:26.149656 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:26.149656 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:26.149657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:26.149761 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:26.149811 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:26.150148 (+ 337us) server_negotiation.cc:299] Negotiation successful
0929 19:05:26.150217 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 19:05:26.150620 22887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:26.141164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.66:49535)
0929 19:05:26.141305 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:26.141364 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:26.141974 (+ 610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:26.142237 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:26.142241 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:26.142909 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:05:26.143080 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:26.143084 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.143767 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.143768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:26.144529 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:26.144530 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.144925 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.144926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:26.145059 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:26.145610 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:05:26.145618 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:05:26.147350 (+ 1732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:05:26.148890 (+ 1540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:26.148897 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:26.148899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:26.149231 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:26.149444 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:26.149446 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:26.149446 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:26.149583 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:26.149823 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:05:26.149830 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:05:26.150001 (+ 171us) client_negotiation.cc:769] Sending connection context
0929 19:05:26.150127 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 19:05:26.150310 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":77,"threads_started":1}
I20260929 19:05:26.151062 22885 heartbeater.cc:347] Connected to a master server at 127.8.158.126:38373
I20260929 19:05:26.151167 22885 heartbeater.cc:464] Registering TS with master...
I20260929 19:05:26.151362 22885 heartbeater.cc:511] Master 127.8.158.126:38373 requested a full tablet report, sending...
I20260929 19:05:26.151721 22560 ts_manager.cc:194] Registered new tserver with Master: bbdc71811c3b4416a687f18009ebc913 (127.8.158.66:38085)
I20260929 19:05:26.152104 22560 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.66@KRBTEST.COM'} at 127.8.158.66:49535
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab.
Entry for principal kudu/127.8.158.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab.
I20260929 19:05:26.167239 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.67:0
--local_ip_for_outbound_sockets=127.8.158.67
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:38373
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab
--principal=kudu/127.8.158.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:34333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:05:26.245508 22890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:05:26.245730 22890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:05:26.245765 22890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:05:26.246896 22890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:05:26.246958 22890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:05:26.247002 22890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.67
I20260929 19:05:26.248418 22890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:34333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab
--principal=kudu/127.8.158.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:38373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.67
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:05:26.248665 22890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:05:26.248883 22890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:05:26.253865 22895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:26.253930 22896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:26.253880 22898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:05:26.254235 22890 server_base.cc:1034] running on GCE node
I20260929 19:05:26.254504 22890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:05:26.254765 22890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:05:26.255940 22890 hybrid_clock.cc:648] HybridClock initialized: now 1790708726255932 us; error 50 us; skew 500 ppm
Sep 29 19:05:26 dist-test-slave-nmsz krb5kdc[22515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708726, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:05:26.258735 22890 init.cc:377] Logged in from keytab as kudu/127.8.158.67@KRBTEST.COM (short username kudu)
I20260929 19:05:26.259179 22890 webserver.cc:460] Webserver started at http://127.8.158.67:39981/ using document root <none> and password file <none>
I20260929 19:05:26.259372 22890 fs_manager.cc:359] Metadata directory not provided
I20260929 19:05:26.259415 22890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:05:26.259505 22890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:05:26.260510 22890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-2/data/instance:
uuid: "fd62ed66bc1040bbbe53171a1d114f7d"
format_stamp: "Formatted at 2026-09-29 19:05:26 on dist-test-slave-nmsz"
server_key: "0182c6a7cad9828c53d09fd544aec07d"
server_key_iv: "45fac18d108d03d341681772878180ff"
server_key_version: "encryptionkey@0"
I20260929 19:05:26.260818 22890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance:
uuid: "fd62ed66bc1040bbbe53171a1d114f7d"
format_stamp: "Formatted at 2026-09-29 19:05:26 on dist-test-slave-nmsz"
server_key: "0182c6a7cad9828c53d09fd544aec07d"
server_key_iv: "45fac18d108d03d341681772878180ff"
server_key_version: "encryptionkey@0"
I20260929 19:05:26.262079 22890 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:05:26.262915 22905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:26.263108 22890 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 19:05:26.263221 22890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-2/wal
uuid: "fd62ed66bc1040bbbe53171a1d114f7d"
format_stamp: "Formatted at 2026-09-29 19:05:26 on dist-test-slave-nmsz"
server_key: "0182c6a7cad9828c53d09fd544aec07d"
server_key_iv: "45fac18d108d03d341681772878180ff"
server_key_version: "encryptionkey@0"
I20260929 19:05:26.263285 22890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:05:26.270592 22890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:05:26.272372 22890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:05:26.272522 22890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:05:26.272725 22890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:05:26.273041 22890 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:05:26.273105 22890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:26.273144 22890 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:05:26.273175 22890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:26.279148 22890 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.67:44703
I20260929 19:05:26.279212 23018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.67:44703 every 8 connection(s)
I20260929 19:05:26.279541 22890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
I20260929 19:05:26.281771 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 22890
I20260929 19:05:26.281890 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance
I20260929 19:05:26.282052 8825 external_mini_cluster.cc:1428] Setting key 2ba8ec8de0f3a8a679fab5ff6e84ea57
Sep 29 19:05:26 dist-test-slave-nmsz krb5kdc[22515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708726, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.67@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:05:26.288612 22600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:26.280397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:54155 (local address 127.8.158.126:38373)
0929 19:05:26.280485 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:26.280488 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:26.280666 (+ 178us) server_negotiation.cc:407] Connection header received
0929 19:05:26.281153 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:26.281154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:26.281165 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:26.281228 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:26.282178 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.282954 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.283909 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.284095 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.284150 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:26.286160 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:26.286161 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:26.286161 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:26.286173 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:26.287422 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:26.287796 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:26.287797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:26.287798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:26.287854 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:26.288063 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:26.288063 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:26.288064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:26.288187 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:26.288241 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:26.288483 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 19:05:26.288552 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 19:05:26.288770 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:26.280381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.67:54155)
0929 19:05:26.280550 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:26.280583 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:26.281050 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:26.281305 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:26.281308 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:26.281838 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:05:26.282073 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:26.282078 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.283038 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.283039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:26.283834 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:26.283836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.284168 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.284169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:26.284260 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:26.284717 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:05:26.284724 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:05:26.286083 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:05:26.287492 (+ 1409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:26.287497 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:26.287499 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:26.287702 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:26.287922 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:26.287923 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:26.287923 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:26.288009 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:26.288243 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:05:26.288246 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:05:26.288351 (+ 105us) client_negotiation.cc:769] Sending connection context
0929 19:05:26.288427 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 19:05:26.288532 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":76,"threads_started":1}
I20260929 19:05:26.289240 23019 heartbeater.cc:347] Connected to a master server at 127.8.158.126:38373
I20260929 19:05:26.289320 23019 heartbeater.cc:464] Registering TS with master...
I20260929 19:05:26.289489 23019 heartbeater.cc:511] Master 127.8.158.126:38373 requested a full tablet report, sending...
I20260929 19:05:26.289836 22560 ts_manager.cc:194] Registered new tserver with Master: fd62ed66bc1040bbbe53171a1d114f7d (127.8.158.67:44703)
I20260929 19:05:26.290210 22560 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.67@KRBTEST.COM'} at 127.8.158.67:54155
I20260929 19:05:26.293715 8825 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:05:26.300858 22600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:26.295016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57758 (local address 127.8.158.126:38373)
0929 19:05:26.295089 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:26.295090 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:26.295177 (+ 87us) server_negotiation.cc:407] Connection header received
0929 19:05:26.295270 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:26.295270 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:26.295279 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:26.295324 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:26.295874 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.296332 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.297040 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.297235 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.297319 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:26.298439 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:26.298440 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:26.298440 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:26.298447 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:26.299637 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:26.299995 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:26.299996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:26.299997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:26.300054 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:26.300287 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:26.300288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:26.300289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:26.300429 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:26.300543 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:26.300744 (+ 201us) server_negotiation.cc:299] Negotiation successful
0929 19:05:26.300800 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 19:05:26.302685 22560 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57758:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:05:26.304180 22560 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:05:26.310401 23027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:26.306345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.67:44703 (local address 127.0.0.1:49914)
0929 19:05:26.306873 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:26.306880 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:26.306934 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:26.307146 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:26.307147 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:26.307149 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:05:26.307198 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:26.307200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.308050 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.308051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:26.309290 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:26.309293 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.310078 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.310079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:26.310156 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:26.310169 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 19:05:26.310233 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 19:05:26.310291 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"mutex_wait_us":2,"thread_start_us":84,"threads_started":1}
I20260929 19:05:26.310458 23028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:26.306344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.66:38085 (local address 127.0.0.1:38644)
0929 19:05:26.306721 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:26.306732 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:26.306798 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:26.307029 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:26.307030 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:26.307032 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:05:26.307074 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:26.307076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.308392 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.308393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:26.309286 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:26.309288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.310191 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.310192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:26.310252 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:26.310254 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 19:05:26.310332 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 19:05:26.310363 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":4,"thread_start_us":53,"threads_started":1}
I20260929 19:05:26.310626 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:26.306396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49914 (local address 127.8.158.67:44703)
0929 19:05:26.306752 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:26.306756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:26.306932 (+ 176us) server_negotiation.cc:407] Connection header received
0929 19:05:26.307004 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:26.307006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:26.307020 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:26.307085 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:05:26.307285 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.307949 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.309386 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.309900 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.309984 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:26.310041 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:26.310512 (+ 471us) server_negotiation.cc:299] Negotiation successful
0929 19:05:26.310546 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":65,"threads_started":1}
I20260929 19:05:26.310657 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:26.306333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:46257 (local address 127.0.0.1:33996)
0929 19:05:26.306495 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:26.306530 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:26.306593 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:26.306985 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:26.306988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:26.306990 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:05:26.307074 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:26.307076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.308359 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.308360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:26.309394 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:26.309396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.310294 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.310295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:26.310385 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:26.310389 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:05:26.310439 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 19:05:26.310595 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":1,"thread_start_us":52,"threads_started":1}
I20260929 19:05:26.310925 23031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:26.306553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38644 (local address 127.8.158.66:38085)
0929 19:05:26.306751 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:26.306756 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:26.306785 (+ 29us) server_negotiation.cc:407] Connection header received
0929 19:05:26.306873 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:26.306875 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:26.306890 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:26.306954 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:05:26.307157 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.308264 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.309368 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.310103 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.310191 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:26.310246 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:26.310341 (+ 95us) server_negotiation.cc:299] Negotiation successful
0929 19:05:26.310380 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":92,"threads_started":1}
I20260929 19:05:26.311247 22953 tablet_service.cc:1467] Processing CreateTablet for tablet 759f70010ac74d198717bcbb5c1914c1 (DEFAULT_TABLE table=test-table [id=80efa031b1ab4ca1ad03bd0e5456c21d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:05:26.311510 22819 tablet_service.cc:1467] Processing CreateTablet for tablet 759f70010ac74d198717bcbb5c1914c1 (DEFAULT_TABLE table=test-table [id=80efa031b1ab4ca1ad03bd0e5456c21d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:05:26.311612 22953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 759f70010ac74d198717bcbb5c1914c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:05:26.311815 22819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 759f70010ac74d198717bcbb5c1914c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:05:26.313984 23033 tablet_bootstrap.cc:492] T 759f70010ac74d198717bcbb5c1914c1 P fd62ed66bc1040bbbe53171a1d114f7d: Bootstrap starting.
I20260929 19:05:26.314042 23032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:26.306554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33996 (local address 127.8.158.65:46257)
0929 19:05:26.306762 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:26.306766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:26.306774 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:05:26.306811 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:26.306813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:26.306830 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:26.306900 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:05:26.307158 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.308259 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.309472 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.310157 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.310214 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:26.310260 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:26.310767 (+ 507us) server_negotiation.cc:299] Negotiation successful
0929 19:05:26.310813 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":93,"threads_started":1}
I20260929 19:05:26.314443 23034 tablet_bootstrap.cc:492] T 759f70010ac74d198717bcbb5c1914c1 P bbdc71811c3b4416a687f18009ebc913: Bootstrap starting.
I20260929 19:05:26.314661 22685 tablet_service.cc:1467] Processing CreateTablet for tablet 759f70010ac74d198717bcbb5c1914c1 (DEFAULT_TABLE table=test-table [id=80efa031b1ab4ca1ad03bd0e5456c21d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:05:26.314963 22685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 759f70010ac74d198717bcbb5c1914c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:05:26.315044 23033 tablet_bootstrap.cc:654] T 759f70010ac74d198717bcbb5c1914c1 P fd62ed66bc1040bbbe53171a1d114f7d: Neither blocks nor log segments found. Creating new log.
I20260929 19:05:26.315071 23034 tablet_bootstrap.cc:654] T 759f70010ac74d198717bcbb5c1914c1 P bbdc71811c3b4416a687f18009ebc913: Neither blocks nor log segments found. Creating new log.
I20260929 19:05:26.315512 23034 log.cc:824] T 759f70010ac74d198717bcbb5c1914c1 P bbdc71811c3b4416a687f18009ebc913: Log is configured to *not* fsync() on all Append() calls
I20260929 19:05:26.315582 23033 log.cc:824] T 759f70010ac74d198717bcbb5c1914c1 P fd62ed66bc1040bbbe53171a1d114f7d: Log is configured to *not* fsync() on all Append() calls
I20260929 19:05:26.316282 23034 tablet_bootstrap.cc:492] T 759f70010ac74d198717bcbb5c1914c1 P bbdc71811c3b4416a687f18009ebc913: No bootstrap required, opened a new log
I20260929 19:05:26.316357 23034 ts_tablet_manager.cc:1397] T 759f70010ac74d198717bcbb5c1914c1 P bbdc71811c3b4416a687f18009ebc913: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:05:26.318096 23033 tablet_bootstrap.cc:492] T 759f70010ac74d198717bcbb5c1914c1 P fd62ed66bc1040bbbe53171a1d114f7d: No bootstrap required, opened a new log
I20260929 19:05:26.318171 23033 ts_tablet_manager.cc:1397] T 759f70010ac74d198717bcbb5c1914c1 P fd62ed66bc1040bbbe53171a1d114f7d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 19:05:26.318349 23037 tablet_bootstrap.cc:492] T 759f70010ac74d198717bcbb5c1914c1 P ffeec491014e4d6b89512c09a168dc9b: Bootstrap starting.
I20260929 19:05:26.318351 23034 raft_consensus.cc:348] T 759f70010ac74d198717bcbb5c1914c1 P bbdc71811c3b4416a687f18009ebc913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffeec491014e4d6b89512c09a168dc9b" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 46257 } } peers { permanent_uuid: "fd62ed66bc1040bbbe53171a1d114f7d" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 44703 } } peers { permanent_uuid: "bbdc71811c3b4416a687f18009ebc913" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38085 } }
I20260929 19:05:26.318539 23034 raft_consensus.cc:374] T 759f70010ac74d198717bcbb5c1914c1 P bbdc71811c3b4416a687f18009ebc913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:05:26.318585 23034 raft_consensus.cc:729] T 759f70010ac74d198717bcbb5c1914c1 P bbdc71811c3b4416a687f18009ebc913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbdc71811c3b4416a687f18009ebc913, State: Initialized, Role: FOLLOWER
I20260929 19:05:26.318694 23034 consensus_queue.cc:260] T 759f70010ac74d198717bcbb5c1914c1 P bbdc71811c3b4416a687f18009ebc913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffeec491014e4d6b89512c09a168dc9b" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 46257 } } peers { permanent_uuid: "fd62ed66bc1040bbbe53171a1d114f7d" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 44703 } } peers { permanent_uuid: "bbdc71811c3b4416a687f18009ebc913" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38085 } }
I20260929 19:05:26.318955 23034 ts_tablet_manager.cc:1428] T 759f70010ac74d198717bcbb5c1914c1 P bbdc71811c3b4416a687f18009ebc913: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 19:05:26.318966 23037 tablet_bootstrap.cc:654] T 759f70010ac74d198717bcbb5c1914c1 P ffeec491014e4d6b89512c09a168dc9b: Neither blocks nor log segments found. Creating new log.
I20260929 19:05:26.319089 22885 heartbeater.cc:503] Master 127.8.158.126:38373 was elected leader, sending a full tablet report...
I20260929 19:05:26.319541 23037 log.cc:824] T 759f70010ac74d198717bcbb5c1914c1 P ffeec491014e4d6b89512c09a168dc9b: Log is configured to *not* fsync() on all Append() calls
I20260929 19:05:26.320443 23037 tablet_bootstrap.cc:492] T 759f70010ac74d198717bcbb5c1914c1 P ffeec491014e4d6b89512c09a168dc9b: No bootstrap required, opened a new log
I20260929 19:05:26.320323 23033 raft_consensus.cc:348] T 759f70010ac74d198717bcbb5c1914c1 P fd62ed66bc1040bbbe53171a1d114f7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffeec491014e4d6b89512c09a168dc9b" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 46257 } } peers { permanent_uuid: "fd62ed66bc1040bbbe53171a1d114f7d" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 44703 } } peers { permanent_uuid: "bbdc71811c3b4416a687f18009ebc913" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38085 } }
I20260929 19:05:26.320487 23033 raft_consensus.cc:374] T 759f70010ac74d198717bcbb5c1914c1 P fd62ed66bc1040bbbe53171a1d114f7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:05:26.320510 23037 ts_tablet_manager.cc:1397] T 759f70010ac74d198717bcbb5c1914c1 P ffeec491014e4d6b89512c09a168dc9b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:05:26.320523 23033 raft_consensus.cc:729] T 759f70010ac74d198717bcbb5c1914c1 P fd62ed66bc1040bbbe53171a1d114f7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd62ed66bc1040bbbe53171a1d114f7d, State: Initialized, Role: FOLLOWER
I20260929 19:05:26.320640 23033 consensus_queue.cc:260] T 759f70010ac74d198717bcbb5c1914c1 P fd62ed66bc1040bbbe53171a1d114f7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffeec491014e4d6b89512c09a168dc9b" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 46257 } } peers { permanent_uuid: "fd62ed66bc1040bbbe53171a1d114f7d" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 44703 } } peers { permanent_uuid: "bbdc71811c3b4416a687f18009ebc913" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38085 } }
I20260929 19:05:26.320896 23033 ts_tablet_manager.cc:1428] T 759f70010ac74d198717bcbb5c1914c1 P fd62ed66bc1040bbbe53171a1d114f7d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 19:05:26.320920 23019 heartbeater.cc:503] Master 127.8.158.126:38373 was elected leader, sending a full tablet report...
I20260929 19:05:26.322481 23037 raft_consensus.cc:348] T 759f70010ac74d198717bcbb5c1914c1 P ffeec491014e4d6b89512c09a168dc9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffeec491014e4d6b89512c09a168dc9b" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 46257 } } peers { permanent_uuid: "fd62ed66bc1040bbbe53171a1d114f7d" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 44703 } } peers { permanent_uuid: "bbdc71811c3b4416a687f18009ebc913" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38085 } }
I20260929 19:05:26.322659 23037 raft_consensus.cc:374] T 759f70010ac74d198717bcbb5c1914c1 P ffeec491014e4d6b89512c09a168dc9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:05:26.322681 23037 raft_consensus.cc:729] T 759f70010ac74d198717bcbb5c1914c1 P ffeec491014e4d6b89512c09a168dc9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffeec491014e4d6b89512c09a168dc9b, State: Initialized, Role: FOLLOWER
I20260929 19:05:26.322808 23037 consensus_queue.cc:260] T 759f70010ac74d198717bcbb5c1914c1 P ffeec491014e4d6b89512c09a168dc9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffeec491014e4d6b89512c09a168dc9b" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 46257 } } peers { permanent_uuid: "fd62ed66bc1040bbbe53171a1d114f7d" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 44703 } } peers { permanent_uuid: "bbdc71811c3b4416a687f18009ebc913" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38085 } }
I20260929 19:05:26.323058 23037 ts_tablet_manager.cc:1428] T 759f70010ac74d198717bcbb5c1914c1 P ffeec491014e4d6b89512c09a168dc9b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 19:05:26.323098 22751 heartbeater.cc:503] Master 127.8.158.126:38373 was elected leader, sending a full tablet report...
I20260929 19:05:26.384743 23038 raft_consensus.cc:482] T 759f70010ac74d198717bcbb5c1914c1 P bbdc71811c3b4416a687f18009ebc913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:05:26.384843 23038 raft_consensus.cc:504] T 759f70010ac74d198717bcbb5c1914c1 P bbdc71811c3b4416a687f18009ebc913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffeec491014e4d6b89512c09a168dc9b" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 46257 } } peers { permanent_uuid: "fd62ed66bc1040bbbe53171a1d114f7d" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 44703 } } peers { permanent_uuid: "bbdc71811c3b4416a687f18009ebc913" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38085 } }
I20260929 19:05:26.385267 23038 leader_election.cc:288] T 759f70010ac74d198717bcbb5c1914c1 P bbdc71811c3b4416a687f18009ebc913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffeec491014e4d6b89512c09a168dc9b (127.8.158.65:46257), fd62ed66bc1040bbbe53171a1d114f7d (127.8.158.67:44703)
I20260929 19:05:26.388199 23032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:26.385472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:48813 (local address 127.8.158.65:46257)
0929 19:05:26.385512 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:26.385514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:26.385522 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:05:26.385620 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:26.385622 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:26.385632 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:26.385715 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:05:26.385909 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.386502 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.387468 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.387851 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.387906 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:26.387913 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:26.388091 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 19:05:26.388132 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 19:05:26.388232 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:26.385576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:52523 (local address 127.8.158.67:44703)
0929 19:05:26.385631 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:26.385633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:26.385695 (+ 62us) server_negotiation.cc:407] Connection header received
0929 19:05:26.385797 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:26.385798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:26.385807 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:26.385865 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:05:26.386027 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.386570 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.387477 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.387861 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.387949 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:26.387957 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:26.388132 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 19:05:26.388168 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 19:05:26.388234 22887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:26.385429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:46257 (local address 127.8.158.66:48813)
0929 19:05:26.385482 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:26.385494 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:26.385537 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:26.385741 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:26.385742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:26.385744 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:05:26.385819 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:26.385821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.386585 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.386586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:26.387380 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:26.387382 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.387933 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.387935 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:26.388021 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:26.388024 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:05:26.388066 (+ 42us) client_negotiation.cc:240] Negotiation successful
0929 19:05:26.388124 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":17}
I20260929 19:05:26.388248 23042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:26.385436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.67:44703 (local address 127.8.158.66:52523)
0929 19:05:26.385629 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:26.385640 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:26.385693 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:26.385890 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:26.385891 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:26.385893 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:05:26.385961 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:26.385963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.386640 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.386643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:26.387384 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:26.387386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.387961 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.387962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:26.388033 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:26.388036 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:05:26.388109 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 19:05:26.388145 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":10,"thread_start_us":90,"threads_started":1}
I20260929 19:05:26.388542 22973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "759f70010ac74d198717bcbb5c1914c1" candidate_uuid: "bbdc71811c3b4416a687f18009ebc913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd62ed66bc1040bbbe53171a1d114f7d" is_pre_election: true
I20260929 19:05:26.388566 22705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "759f70010ac74d198717bcbb5c1914c1" candidate_uuid: "bbdc71811c3b4416a687f18009ebc913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffeec491014e4d6b89512c09a168dc9b" is_pre_election: true
I20260929 19:05:26.388684 22973 raft_consensus.cc:2454] T 759f70010ac74d198717bcbb5c1914c1 P fd62ed66bc1040bbbe53171a1d114f7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbdc71811c3b4416a687f18009ebc913 in term 0.
I20260929 19:05:26.388725 22705 raft_consensus.cc:2454] T 759f70010ac74d198717bcbb5c1914c1 P ffeec491014e4d6b89512c09a168dc9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbdc71811c3b4416a687f18009ebc913 in term 0.
I20260929 19:05:26.388895 22774 leader_election.cc:302] T 759f70010ac74d198717bcbb5c1914c1 P bbdc71811c3b4416a687f18009ebc913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbdc71811c3b4416a687f18009ebc913, fd62ed66bc1040bbbe53171a1d114f7d; no voters:
I20260929 19:05:26.389047 23038 raft_consensus.cc:2781] T 759f70010ac74d198717bcbb5c1914c1 P bbdc71811c3b4416a687f18009ebc913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:05:26.389102 23038 raft_consensus.cc:482] T 759f70010ac74d198717bcbb5c1914c1 P bbdc71811c3b4416a687f18009ebc913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:05:26.389125 23038 raft_consensus.cc:3037] T 759f70010ac74d198717bcbb5c1914c1 P bbdc71811c3b4416a687f18009ebc913 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:05:26.389786 23038 raft_consensus.cc:504] T 759f70010ac74d198717bcbb5c1914c1 P bbdc71811c3b4416a687f18009ebc913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffeec491014e4d6b89512c09a168dc9b" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 46257 } } peers { permanent_uuid: "fd62ed66bc1040bbbe53171a1d114f7d" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 44703 } } peers { permanent_uuid: "bbdc71811c3b4416a687f18009ebc913" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38085 } }
I20260929 19:05:26.389909 23038 leader_election.cc:288] T 759f70010ac74d198717bcbb5c1914c1 P bbdc71811c3b4416a687f18009ebc913 [CANDIDATE]: Term 1 election: Requested vote from peers ffeec491014e4d6b89512c09a168dc9b (127.8.158.65:46257), fd62ed66bc1040bbbe53171a1d114f7d (127.8.158.67:44703)
I20260929 19:05:26.390050 22705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "759f70010ac74d198717bcbb5c1914c1" candidate_uuid: "bbdc71811c3b4416a687f18009ebc913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffeec491014e4d6b89512c09a168dc9b"
I20260929 19:05:26.390122 22705 raft_consensus.cc:3037] T 759f70010ac74d198717bcbb5c1914c1 P ffeec491014e4d6b89512c09a168dc9b [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:05:26.390142 22973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "759f70010ac74d198717bcbb5c1914c1" candidate_uuid: "bbdc71811c3b4416a687f18009ebc913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd62ed66bc1040bbbe53171a1d114f7d"
I20260929 19:05:26.390252 22973 raft_consensus.cc:3037] T 759f70010ac74d198717bcbb5c1914c1 P fd62ed66bc1040bbbe53171a1d114f7d [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:05:26.390832 22705 raft_consensus.cc:2454] T 759f70010ac74d198717bcbb5c1914c1 P ffeec491014e4d6b89512c09a168dc9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbdc71811c3b4416a687f18009ebc913 in term 1.
I20260929 19:05:26.391007 22775 leader_election.cc:302] T 759f70010ac74d198717bcbb5c1914c1 P bbdc71811c3b4416a687f18009ebc913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbdc71811c3b4416a687f18009ebc913, ffeec491014e4d6b89512c09a168dc9b; no voters:
I20260929 19:05:26.391036 22973 raft_consensus.cc:2454] T 759f70010ac74d198717bcbb5c1914c1 P fd62ed66bc1040bbbe53171a1d114f7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbdc71811c3b4416a687f18009ebc913 in term 1.
W20260929 19:05:26.391016 22886 tablet.cc:2369] T 759f70010ac74d198717bcbb5c1914c1 P bbdc71811c3b4416a687f18009ebc913: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:05:26.391112 23038 raft_consensus.cc:2781] T 759f70010ac74d198717bcbb5c1914c1 P bbdc71811c3b4416a687f18009ebc913 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:05:26.391248 23038 raft_consensus.cc:686] T 759f70010ac74d198717bcbb5c1914c1 P bbdc71811c3b4416a687f18009ebc913 [term 1 LEADER]: Becoming Leader. State: Replica: bbdc71811c3b4416a687f18009ebc913, State: Running, Role: LEADER
I20260929 19:05:26.391351 23038 consensus_queue.cc:237] T 759f70010ac74d198717bcbb5c1914c1 P bbdc71811c3b4416a687f18009ebc913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffeec491014e4d6b89512c09a168dc9b" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 46257 } } peers { permanent_uuid: "fd62ed66bc1040bbbe53171a1d114f7d" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 44703 } } peers { permanent_uuid: "bbdc71811c3b4416a687f18009ebc913" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38085 } }
I20260929 19:05:26.392055 22560 catalog_manager.cc:5495] T 759f70010ac74d198717bcbb5c1914c1 P bbdc71811c3b4416a687f18009ebc913 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbdc71811c3b4416a687f18009ebc913 (127.8.158.66). New cstate: current_term: 1 leader_uuid: "bbdc71811c3b4416a687f18009ebc913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffeec491014e4d6b89512c09a168dc9b" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 46257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd62ed66bc1040bbbe53171a1d114f7d" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 44703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbdc71811c3b4416a687f18009ebc913" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38085 } health_report { overall_health: HEALTHY } } }
I20260929 19:05:26.400899 23031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:26.398569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38660 (local address 127.8.158.66:38085)
0929 19:05:26.398625 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:26.398627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:26.398635 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:05:26.398666 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:26.398667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:26.398683 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:26.398765 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:05:26.398920 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.399352 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.400095 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.400277 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.400355 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:26.400524 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:05:26.400637 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:05:26.400714 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:26.400739 (+ 25us) server_negotiation.cc:299] Negotiation successful
0929 19:05:26.400756 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 19:05:26.402381 22705 raft_consensus.cc:1261] T 759f70010ac74d198717bcbb5c1914c1 P ffeec491014e4d6b89512c09a168dc9b [term 1 FOLLOWER]: Refusing update from remote peer bbdc71811c3b4416a687f18009ebc913: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:05:26.402424 22973 raft_consensus.cc:1261] T 759f70010ac74d198717bcbb5c1914c1 P fd62ed66bc1040bbbe53171a1d114f7d [term 1 FOLLOWER]: Refusing update from remote peer bbdc71811c3b4416a687f18009ebc913: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:05:26.402709 23043 consensus_queue.cc:1035] T 759f70010ac74d198717bcbb5c1914c1 P bbdc71811c3b4416a687f18009ebc913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffeec491014e4d6b89512c09a168dc9b" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 46257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:05:26.402837 23038 consensus_queue.cc:1035] T 759f70010ac74d198717bcbb5c1914c1 P bbdc71811c3b4416a687f18009ebc913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd62ed66bc1040bbbe53171a1d114f7d" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 44703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:05:26.404809 23046 mvcc.cc:205] Tried to move back new op lower bound from 7334742943342047232 to 7334742943299588096. Current Snapshot: MvccSnapshot[applied={T|T < 7334742943342047232}]
I20260929 19:05:26.405100 23047 mvcc.cc:205] Tried to move back new op lower bound from 7334742943342047232 to 7334742943299588096. Current Snapshot: MvccSnapshot[applied={T|T < 7334742943342047232}]
I20260929 19:05:26.407284 22559 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:05:26.407377 22559 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:05:26.408102 22559 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054: Sending DeleteTablet for 3 replicas of tablet 759f70010ac74d198717bcbb5c1914c1
I20260929 19:05:26.408429 22685 tablet_service.cc:1514] Processing DeleteTablet for tablet 759f70010ac74d198717bcbb5c1914c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:05:26 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:33996
I20260929 19:05:26.408591 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 22622
I20260929 19:05:26.408600 22819 tablet_service.cc:1514] Processing DeleteTablet for tablet 759f70010ac74d198717bcbb5c1914c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:05:26 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:38644
I20260929 19:05:26.408655 22953 tablet_service.cc:1514] Processing DeleteTablet for tablet 759f70010ac74d198717bcbb5c1914c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:05:26 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:49914
W20260929 19:05:26.413658 22547 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.158.65:46257 (error 108)
I20260929 19:05:26.413945 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 22756
I20260929 19:05:26.414075 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:26.413956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:46257 (local address 127.0.0.1:33998)
0929 19:05:26.414010 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:26.414043 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.65:46257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 19:05:26.414641 22547 catalog_manager.cc:4584] TS ffeec491014e4d6b89512c09a168dc9b (127.8.158.65:46257): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 759f70010ac74d198717bcbb5c1914c1: Network error: Client connection negotiation failed: client connection to 127.8.158.65:46257: connect: Connection refused (error 111)
I20260929 19:05:26.414986 23058 tablet_replica.cc:330] T 759f70010ac74d198717bcbb5c1914c1 P fd62ed66bc1040bbbe53171a1d114f7d: stopping tablet replica
I20260929 19:05:26.415056 23058 raft_consensus.cc:2229] T 759f70010ac74d198717bcbb5c1914c1 P fd62ed66bc1040bbbe53171a1d114f7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:05:26.415120 23058 raft_consensus.cc:2258] T 759f70010ac74d198717bcbb5c1914c1 P fd62ed66bc1040bbbe53171a1d114f7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:05:26.415547 23058 ts_tablet_manager.cc:1905] T 759f70010ac74d198717bcbb5c1914c1 P fd62ed66bc1040bbbe53171a1d114f7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:05:26.417042 23058 ts_tablet_manager.cc:1918] T 759f70010ac74d198717bcbb5c1914c1 P fd62ed66bc1040bbbe53171a1d114f7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:05:26.417128 23058 log.cc:1196] T 759f70010ac74d198717bcbb5c1914c1 P fd62ed66bc1040bbbe53171a1d114f7d: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-2/wal/wals/759f70010ac74d198717bcbb5c1914c1
I20260929 19:05:26.417410 23058 ts_tablet_manager.cc:1939] T 759f70010ac74d198717bcbb5c1914c1 P fd62ed66bc1040bbbe53171a1d114f7d: Deleting consensus metadata
I20260929 19:05:26.417809 22546 catalog_manager.cc:4841] TS fd62ed66bc1040bbbe53171a1d114f7d (127.8.158.67:44703): tablet 759f70010ac74d198717bcbb5c1914c1 (table test-table [id=80efa031b1ab4ca1ad03bd0e5456c21d]) successfully deleted
I20260929 19:05:26.419057 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 22890
I20260929 19:05:26.419200 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:26.419061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.66:38085 (local address 127.0.0.1:38674)
0929 19:05:26.419120 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:26.419169 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.66:38085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 19:05:26.424194 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 22528
I20260929 19:05:26.429301 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:38373
--webserver_interface=127.8.158.126
--webserver_port=33551
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:34333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:38373
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:05:26.507833 23060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:05:26.508041 23060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:05:26.508064 23060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:05:26.509197 23060 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:05:26.509240 23060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:05:26.509255 23060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:05:26.509268 23060 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:05:26.509330 23060 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:05:26.510787 23060 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:34333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:38373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:38373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.158.126
--webserver_port=33551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:05:26.511019 23060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:05:26.511260 23060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:05:26.519515 23069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:26.519543 23067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:26.519548 23066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:05:26.519758 23060 server_base.cc:1034] running on GCE node
I20260929 19:05:26.520025 23060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:05:26.520349 23060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:05:26.521492 23060 hybrid_clock.cc:648] HybridClock initialized: now 1790708726521485 us; error 29 us; skew 500 ppm
Sep 29 19:05:26 dist-test-slave-nmsz krb5kdc[22515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708726, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:05:26.523929 23060 init.cc:377] Logged in from keytab as kudu/127.8.158.126@KRBTEST.COM (short username kudu)
I20260929 19:05:26.524418 23060 webserver.cc:460] Webserver started at http://127.8.158.126:33551/ using document root <none> and password file <none>
I20260929 19:05:26.524646 23060 fs_manager.cc:359] Metadata directory not provided
I20260929 19:05:26.524726 23060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:05:26.526073 23060 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:05:26.526809 23076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:26.526975 23060 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:26.527078 23060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "7181e789b0c24f22b65f8fdd3e231054"
format_stamp: "Formatted at 2026-09-29 19:05:25 on dist-test-slave-nmsz"
server_key: "327656b0ce57b1499e5606bdaeaa9ab5"
server_key_iv: "225f646728bcf3f8368a57734be38341"
server_key_version: "encryptionkey@0"
I20260929 19:05:26.527469 23060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:05:26.558475 23060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:05:26.560398 23060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:05:26.560566 23060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:05:26.564628 23060 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.126:38373
I20260929 19:05:26.564659 23128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.126:38373 every 8 connection(s)
I20260929 19:05:26.565026 23060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:05:26.566563 23129 sys_catalog.cc:263] Verifying existing consensus state
I20260929 19:05:26.567211 23129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054: Bootstrap starting.
I20260929 19:05:26.574030 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 23060
I20260929 19:05:26.574385 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.65:46257
--local_ip_for_outbound_sockets=127.8.158.65
--tserver_master_addrs=127.8.158.126:38373
--webserver_port=43355
--webserver_interface=127.8.158.65
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab
--principal=kudu/127.8.158.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:34333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:05:26.575618 23129 log.cc:824] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054: Log is configured to *not* fsync() on all Append() calls
I20260929 19:05:26.577491 23129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 19:05:26.577847 23129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054: Bootstrap complete.
I20260929 19:05:26.580118 23129 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7181e789b0c24f22b65f8fdd3e231054" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 38373 } }
I20260929 19:05:26.580328 23129 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7181e789b0c24f22b65f8fdd3e231054, State: Initialized, Role: FOLLOWER
I20260929 19:05:26.580492 23129 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7181e789b0c24f22b65f8fdd3e231054" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 38373 } }
I20260929 19:05:26.580552 23129 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:05:26.580582 23129 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:05:26.580650 23129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [term 1 FOLLOWER]: Advancing to term 2
I20260929 19:05:26.581617 23129 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7181e789b0c24f22b65f8fdd3e231054" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 38373 } }
I20260929 19:05:26.581702 23129 leader_election.cc:302] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7181e789b0c24f22b65f8fdd3e231054; no voters:
I20260929 19:05:26.581877 23129 leader_election.cc:288] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 19:05:26.581941 23133 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 19:05:26.582087 23133 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [term 2 LEADER]: Becoming Leader. State: Replica: 7181e789b0c24f22b65f8fdd3e231054, State: Running, Role: LEADER
I20260929 19:05:26.582144 23129 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:05:26.582198 23133 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7181e789b0c24f22b65f8fdd3e231054" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 38373 } }
I20260929 19:05:26.582605 23134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7181e789b0c24f22b65f8fdd3e231054" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7181e789b0c24f22b65f8fdd3e231054" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 38373 } } }
I20260929 19:05:26.582700 23134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [sys.catalog]: This master's current role is: LEADER
I20260929 19:05:26.582911 23135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7181e789b0c24f22b65f8fdd3e231054. Latest consensus state: current_term: 2 leader_uuid: "7181e789b0c24f22b65f8fdd3e231054" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7181e789b0c24f22b65f8fdd3e231054" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 38373 } } }
I20260929 19:05:26.583034 23135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [sys.catalog]: This master's current role is: LEADER
I20260929 19:05:26.583079 23142 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:05:26.583894 23142 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:05:26.584005 23142 catalog_manager.cc:1259] Loaded cluster ID: 1cb79e5cf82d40e78d3c5340b60f0ca3
I20260929 19:05:26.584033 23142 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:05:26.584565 23142 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 19:05:26.589634 23142 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 19:05:26.592039 23142 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:05:26.592211 23142 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054: Loaded TSK: 0
I20260929 19:05:26.592407 23142 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 19:05:26.657018 23131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:05:26.657203 23131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:05:26.657226 23131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:05:26.658403 23131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:05:26.658442 23131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:05:26.658468 23131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.65
I20260929 19:05:26.659890 23131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:34333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab
--principal=kudu/127.8.158.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.65:46257
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.158.65
--webserver_port=43355
--tserver_master_addrs=127.8.158.126:38373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.65
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:05:26.660111 23131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:05:26.660348 23131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:05:26.666195 23157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:26.666189 23156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:05:26.666422 23131 server_base.cc:1034] running on GCE node
W20260929 19:05:26.666531 23159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:05:26.666740 23131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:05:26.666972 23131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:05:26.668140 23131 hybrid_clock.cc:648] HybridClock initialized: now 1790708726668127 us; error 29 us; skew 500 ppm
Sep 29 19:05:26 dist-test-slave-nmsz krb5kdc[22515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708726, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:05:26.670800 23131 init.cc:377] Logged in from keytab as kudu/127.8.158.65@KRBTEST.COM (short username kudu)
I20260929 19:05:26.671324 23131 webserver.cc:460] Webserver started at http://127.8.158.65:43355/ using document root <none> and password file <none>
I20260929 19:05:26.671578 23131 fs_manager.cc:359] Metadata directory not provided
I20260929 19:05:26.671650 23131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:05:26.673316 23131 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:05:26.674085 23166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:26.674360 23131 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:05:26.674455 23131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-0/wal
uuid: "ffeec491014e4d6b89512c09a168dc9b"
format_stamp: "Formatted at 2026-09-29 19:05:25 on dist-test-slave-nmsz"
server_key: "c303953b3fd52bb2c7c754497c42d7fa"
server_key_iv: "bc0a2b9477c4cca25bed3ec569d2e990"
server_key_version: "encryptionkey@0"
I20260929 19:05:26.674878 23131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:05:26.701934 23131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:05:26.704478 23131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:05:26.704633 23131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:05:26.704866 23131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:05:26.705376 23173 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 19:05:26.706897 23131 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 19:05:26.706965 23131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260929 19:05:26.707000 23131 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 19:05:26.707882 23131 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 19:05:26.707927 23131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260929 19:05:26.708021 23173 tablet_bootstrap.cc:492] T 759f70010ac74d198717bcbb5c1914c1 P ffeec491014e4d6b89512c09a168dc9b: Bootstrap starting.
I20260929 19:05:26.714341 23131 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.65:46257
I20260929 19:05:26.714721 23131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
I20260929 19:05:26.718091 23173 log.cc:824] T 759f70010ac74d198717bcbb5c1914c1 P ffeec491014e4d6b89512c09a168dc9b: Log is configured to *not* fsync() on all Append() calls
I20260929 19:05:26.718852 23280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.65:46257 every 8 connection(s)
I20260929 19:05:26.719692 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 23131
I20260929 19:05:26.719982 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.66:38085
--local_ip_for_outbound_sockets=127.8.158.66
--tserver_master_addrs=127.8.158.126:38373
--webserver_port=32849
--webserver_interface=127.8.158.66
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab
--principal=kudu/127.8.158.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:34333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:05:26.720721 23173 tablet_bootstrap.cc:492] T 759f70010ac74d198717bcbb5c1914c1 P ffeec491014e4d6b89512c09a168dc9b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 19:05:26.721019 23173 tablet_bootstrap.cc:492] T 759f70010ac74d198717bcbb5c1914c1 P ffeec491014e4d6b89512c09a168dc9b: Bootstrap complete.
I20260929 19:05:26.723326 23173 ts_tablet_manager.cc:1397] T 759f70010ac74d198717bcbb5c1914c1 P ffeec491014e4d6b89512c09a168dc9b: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.005s
I20260929 19:05:26.724679 23173 raft_consensus.cc:348] T 759f70010ac74d198717bcbb5c1914c1 P ffeec491014e4d6b89512c09a168dc9b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffeec491014e4d6b89512c09a168dc9b" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 46257 } } peers { permanent_uuid: "fd62ed66bc1040bbbe53171a1d114f7d" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 44703 } } peers { permanent_uuid: "bbdc71811c3b4416a687f18009ebc913" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38085 } }
I20260929 19:05:26.724937 23173 raft_consensus.cc:729] T 759f70010ac74d198717bcbb5c1914c1 P ffeec491014e4d6b89512c09a168dc9b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffeec491014e4d6b89512c09a168dc9b, State: Initialized, Role: FOLLOWER
I20260929 19:05:26.725064 23173 consensus_queue.cc:260] T 759f70010ac74d198717bcbb5c1914c1 P ffeec491014e4d6b89512c09a168dc9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffeec491014e4d6b89512c09a168dc9b" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 46257 } } peers { permanent_uuid: "fd62ed66bc1040bbbe53171a1d114f7d" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 44703 } } peers { permanent_uuid: "bbdc71811c3b4416a687f18009ebc913" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38085 } }
I20260929 19:05:26.725342 23173 ts_tablet_manager.cc:1428] T 759f70010ac74d198717bcbb5c1914c1 P ffeec491014e4d6b89512c09a168dc9b: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
Sep 29 19:05:26 dist-test-slave-nmsz krb5kdc[22515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708726, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:05:26.731478 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:26.719270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.65:46491)
0929 19:05:26.719489 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:26.719549 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:26.720710 (+ 1161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:26.721307 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:26.721311 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:26.721703 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:05:26.721971 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:26.721977 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.723272 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.723273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:26.724469 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:26.724472 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.725190 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.725191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:26.725347 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:26.725931 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:05:26.725939 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:05:26.727525 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:05:26.729782 (+ 2257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:26.729787 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:26.729789 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:26.730019 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:26.730390 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:26.730390 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:26.730391 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:26.730489 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:26.730963 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:05:26.730967 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:05:26.731101 (+ 134us) client_negotiation.cc:769] Sending connection context
0929 19:05:26.731214 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 19:05:26.731339 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":120,"threads_started":1}
I20260929 19:05:26.731665 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:26.719416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:46491 (local address 127.8.158.126:38373)
0929 19:05:26.719600 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:26.719609 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:26.719645 (+ 36us) server_negotiation.cc:407] Connection header received
0929 19:05:26.720991 (+ 1346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:26.720999 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:26.721115 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:26.721271 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:26.722106 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.723173 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.724554 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.724956 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.725062 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:26.727631 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:26.727635 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:26.727636 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:26.727647 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:26.729684 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:26.730148 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:26.730152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:26.730152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:26.730284 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:26.730564 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:26.730565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:26.730566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:26.730880 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:26.730993 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:26.731282 (+ 289us) server_negotiation.cc:299] Negotiation successful
0929 19:05:26.731432 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":78,"threads_started":1}
I20260929 19:05:26.732273 23281 heartbeater.cc:347] Connected to a master server at 127.8.158.126:38373
I20260929 19:05:26.732378 23281 heartbeater.cc:464] Registering TS with master...
I20260929 19:05:26.732549 23281 heartbeater.cc:511] Master 127.8.158.126:38373 requested a full tablet report, sending...
I20260929 19:05:26.733230 23093 ts_manager.cc:194] Registered new tserver with Master: ffeec491014e4d6b89512c09a168dc9b (127.8.158.65:46257)
I20260929 19:05:26.734215 23093 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.65@KRBTEST.COM'} at 127.8.158.65:46491
I20260929 19:05:26.735000 23281 heartbeater.cc:503] Master 127.8.158.126:38373 was elected leader, sending a full tablet report...
I20260929 19:05:26.738291 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:26.733826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34014 (local address 127.8.158.65:46257)
0929 19:05:26.733958 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:26.733962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:26.734626 (+ 664us) server_negotiation.cc:407] Connection header received
0929 19:05:26.735236 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:26.735239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:26.735254 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:26.735335 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:05:26.735484 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.736109 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.737533 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.737958 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.738015 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:26.738045 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:26.738219 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 19:05:26.738246 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":71,"threads_started":1}
I20260929 19:05:26.738350 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:26.733762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:46257 (local address 127.0.0.1:34014)
0929 19:05:26.734523 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:26.734554 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:26.735132 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:26.735344 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:26.735347 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:26.735350 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:05:26.735406 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:26.735408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.736186 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.736188 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:26.737451 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:26.737453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.738036 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.738037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:26.738118 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:26.738131 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 19:05:26.738208 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 19:05:26.738258 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"thread_start_us":77,"threads_started":1}
I20260929 19:05:26.738883 23214 tablet_service.cc:1514] Processing DeleteTablet for tablet 759f70010ac74d198717bcbb5c1914c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:05:26 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:34014
I20260929 19:05:26.739323 23290 tablet_replica.cc:330] stopping tablet replica
I20260929 19:05:26.739421 23290 raft_consensus.cc:2229] T 759f70010ac74d198717bcbb5c1914c1 P ffeec491014e4d6b89512c09a168dc9b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:05:26.739483 23290 raft_consensus.cc:2258] T 759f70010ac74d198717bcbb5c1914c1 P ffeec491014e4d6b89512c09a168dc9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:05:26.739782 23290 ts_tablet_manager.cc:1905] T 759f70010ac74d198717bcbb5c1914c1 P ffeec491014e4d6b89512c09a168dc9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:05:26.742386 23290 ts_tablet_manager.cc:1918] T 759f70010ac74d198717bcbb5c1914c1 P ffeec491014e4d6b89512c09a168dc9b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:05:26.742458 23290 log.cc:1196] T 759f70010ac74d198717bcbb5c1914c1 P ffeec491014e4d6b89512c09a168dc9b: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-0/wal/wals/759f70010ac74d198717bcbb5c1914c1
I20260929 19:05:26.742692 23290 ts_tablet_manager.cc:1939] T 759f70010ac74d198717bcbb5c1914c1 P ffeec491014e4d6b89512c09a168dc9b: Deleting consensus metadata
I20260929 19:05:26.743172 23080 catalog_manager.cc:4841] TS ffeec491014e4d6b89512c09a168dc9b (127.8.158.65:46257): tablet 759f70010ac74d198717bcbb5c1914c1 (table test-table [id=80efa031b1ab4ca1ad03bd0e5456c21d]) successfully deleted
W20260929 19:05:26.808010 23286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:05:26.808202 23286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:05:26.808224 23286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:05:26.809363 23286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:05:26.809403 23286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:05:26.809429 23286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.66
I20260929 19:05:26.810856 23286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:34333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab
--principal=kudu/127.8.158.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.66:38085
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.158.66
--webserver_port=32849
--tserver_master_addrs=127.8.158.126:38373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.66
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:05:26.811069 23286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:05:26.811345 23286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:05:26.816092 23298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:26.816120 23295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:26.816102 23296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:05:26.816390 23286 server_base.cc:1034] running on GCE node
I20260929 19:05:26.816581 23286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:05:26.816823 23286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:05:26.817966 23286 hybrid_clock.cc:648] HybridClock initialized: now 1790708726817967 us; error 40 us; skew 500 ppm
Sep 29 19:05:26 dist-test-slave-nmsz krb5kdc[22515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708726, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:05:26.820401 23286 init.cc:377] Logged in from keytab as kudu/127.8.158.66@KRBTEST.COM (short username kudu)
I20260929 19:05:26.820799 23286 webserver.cc:460] Webserver started at http://127.8.158.66:32849/ using document root <none> and password file <none>
I20260929 19:05:26.821022 23286 fs_manager.cc:359] Metadata directory not provided
I20260929 19:05:26.821099 23286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:05:26.822484 23286 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:05:26.823201 23305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:26.823413 23286 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:05:26.823510 23286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-1/wal
uuid: "bbdc71811c3b4416a687f18009ebc913"
format_stamp: "Formatted at 2026-09-29 19:05:26 on dist-test-slave-nmsz"
server_key: "860585501b14d580da7a42459a7852e0"
server_key_iv: "1687d7ca24be53ddd9e1f1c6b371f095"
server_key_version: "encryptionkey@0"
I20260929 19:05:26.823822 23286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:05:26.864651 23286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:05:26.866657 23286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:05:26.866813 23286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:05:26.867048 23286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:05:26.867465 23312 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 19:05:26.869211 23286 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 19:05:26.869297 23286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260929 19:05:26.869346 23286 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 19:05:26.869942 23286 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 19:05:26.870005 23286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260929 19:05:26.870092 23312 tablet_bootstrap.cc:492] T 759f70010ac74d198717bcbb5c1914c1 P bbdc71811c3b4416a687f18009ebc913: Bootstrap starting.
I20260929 19:05:26.876673 23286 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.66:38085
I20260929 19:05:26.876763 23419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.66:38085 every 8 connection(s)
I20260929 19:05:26.877069 23286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
I20260929 19:05:26.878762 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 23286
I20260929 19:05:26.879109 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.67:44703
--local_ip_for_outbound_sockets=127.8.158.67
--tserver_master_addrs=127.8.158.126:38373
--webserver_port=39981
--webserver_interface=127.8.158.67
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab
--principal=kudu/127.8.158.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:34333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:05:26.880050 23312 log.cc:824] T 759f70010ac74d198717bcbb5c1914c1 P bbdc71811c3b4416a687f18009ebc913: Log is configured to *not* fsync() on all Append() calls
I20260929 19:05:26.883482 23312 tablet_bootstrap.cc:492] T 759f70010ac74d198717bcbb5c1914c1 P bbdc71811c3b4416a687f18009ebc913: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
Sep 29 19:05:26 dist-test-slave-nmsz krb5kdc[22515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708726, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.66@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:05:26.883771 23312 tablet_bootstrap.cc:492] T 759f70010ac74d198717bcbb5c1914c1 P bbdc71811c3b4416a687f18009ebc913: Bootstrap complete.
I20260929 19:05:26.884861 23312 ts_tablet_manager.cc:1397] T 759f70010ac74d198717bcbb5c1914c1 P bbdc71811c3b4416a687f18009ebc913: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.004s
I20260929 19:05:26.886381 23312 raft_consensus.cc:348] T 759f70010ac74d198717bcbb5c1914c1 P bbdc71811c3b4416a687f18009ebc913 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffeec491014e4d6b89512c09a168dc9b" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 46257 } } peers { permanent_uuid: "fd62ed66bc1040bbbe53171a1d114f7d" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 44703 } } peers { permanent_uuid: "bbdc71811c3b4416a687f18009ebc913" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38085 } }
I20260929 19:05:26.886713 23312 raft_consensus.cc:729] T 759f70010ac74d198717bcbb5c1914c1 P bbdc71811c3b4416a687f18009ebc913 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbdc71811c3b4416a687f18009ebc913, State: Initialized, Role: FOLLOWER
I20260929 19:05:26.886845 23312 consensus_queue.cc:260] T 759f70010ac74d198717bcbb5c1914c1 P bbdc71811c3b4416a687f18009ebc913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffeec491014e4d6b89512c09a168dc9b" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 46257 } } peers { permanent_uuid: "fd62ed66bc1040bbbe53171a1d114f7d" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 44703 } } peers { permanent_uuid: "bbdc71811c3b4416a687f18009ebc913" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38085 } }
I20260929 19:05:26.887104 23312 ts_tablet_manager.cc:1428] T 759f70010ac74d198717bcbb5c1914c1 P bbdc71811c3b4416a687f18009ebc913: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20260929 19:05:26.887259 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:26.877807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:38031 (local address 127.8.158.126:38373)
0929 19:05:26.877846 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:26.877848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:26.878096 (+ 248us) server_negotiation.cc:407] Connection header received
0929 19:05:26.878670 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:26.878672 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:26.878690 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:26.878740 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:26.879572 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.880283 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.881398 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.881652 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.881718 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:26.884206 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:26.884208 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:26.884209 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:26.884217 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:26.885784 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:26.886175 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:26.886177 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:26.886178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:26.886246 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:26.886545 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:26.886546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:26.886547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:26.886690 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:26.886770 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:26.887067 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 19:05:26.887162 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 19:05:26.887380 23422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:26.877782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.66:38031)
0929 19:05:26.877960 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:26.877997 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:26.878555 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:26.878793 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:26.878797 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:26.879224 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:05:26.879475 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:26.879480 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.880399 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.880400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:26.881308 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:26.881310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.881766 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.881767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:26.881906 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:26.882486 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:05:26.882496 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:05:26.884117 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:05:26.885870 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:26.885876 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:26.885877 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:26.886087 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:26.886371 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:26.886372 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:26.886372 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:26.886477 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:26.886771 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:05:26.886777 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:05:26.886908 (+ 131us) client_negotiation.cc:769] Sending connection context
0929 19:05:26.887025 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 19:05:26.887173 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":107,"threads_started":1}
I20260929 19:05:26.888047 23420 heartbeater.cc:347] Connected to a master server at 127.8.158.126:38373
I20260929 19:05:26.888142 23420 heartbeater.cc:464] Registering TS with master...
I20260929 19:05:26.888317 23420 heartbeater.cc:511] Master 127.8.158.126:38373 requested a full tablet report, sending...
I20260929 19:05:26.888859 23093 ts_manager.cc:194] Registered new tserver with Master: bbdc71811c3b4416a687f18009ebc913 (127.8.158.66:38085)
I20260929 19:05:26.889544 23093 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.66@KRBTEST.COM'} at 127.8.158.66:38031
I20260929 19:05:26.890431 23420 heartbeater.cc:503] Master 127.8.158.126:38373 was elected leader, sending a full tablet report...
Sep 29 19:05:26 dist-test-slave-nmsz krb5kdc[22515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708726, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for kudu/127.8.158.66@KRBTEST.COM
I20260929 19:05:26.896828 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:26.889258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.66:38085 (local address 127.0.0.1:38688)
0929 19:05:26.889312 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:26.889322 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:26.889368 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:26.889619 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:26.889620 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:26.889977 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:05:26.890009 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:26.890011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.891024 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.891025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:26.891506 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:26.891508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.891973 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.891974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:26.892027 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:26.892524 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:05:26.892541 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:05:26.893862 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:05:26.895729 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:26.895736 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:26.895738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:26.895930 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:26.896155 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:26.896156 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:26.896157 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:26.896216 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:26.896494 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:05:26.896500 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:05:26.896547 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 19:05:26.896640 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 19:05:26.896733 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 19:05:26.896849 23426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:26.889310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38688 (local address 127.8.158.66:38085)
0929 19:05:26.889458 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:26.889462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:26.889470 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:05:26.889505 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:26.889507 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:26.889519 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:26.889651 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:26.890102 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.890938 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.891601 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:26.891892 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:26.891974 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:26.893953 (+ 1979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:26.893960 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:26.893960 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:26.893966 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:26.895629 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:26.896028 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:26.896030 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:26.896031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:26.896086 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:26.896289 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:26.896290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:26.896290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:26.896420 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:26.896511 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:26.896673 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 19:05:26.896757 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":74,"threads_started":1}
I20260929 19:05:26.897176 23354 tablet_service.cc:1514] Processing DeleteTablet for tablet 759f70010ac74d198717bcbb5c1914c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:05:26 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:38688
I20260929 19:05:26.897632 23427 tablet_replica.cc:330] stopping tablet replica
I20260929 19:05:26.897738 23427 raft_consensus.cc:2229] T 759f70010ac74d198717bcbb5c1914c1 P bbdc71811c3b4416a687f18009ebc913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:05:26.897797 23427 raft_consensus.cc:2258] T 759f70010ac74d198717bcbb5c1914c1 P bbdc71811c3b4416a687f18009ebc913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:05:26.898057 23427 ts_tablet_manager.cc:1905] T 759f70010ac74d198717bcbb5c1914c1 P bbdc71811c3b4416a687f18009ebc913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:05:26.900348 23427 ts_tablet_manager.cc:1918] T 759f70010ac74d198717bcbb5c1914c1 P bbdc71811c3b4416a687f18009ebc913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:05:26.900421 23427 log.cc:1196] T 759f70010ac74d198717bcbb5c1914c1 P bbdc71811c3b4416a687f18009ebc913: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-1/wal/wals/759f70010ac74d198717bcbb5c1914c1
I20260929 19:05:26.900635 23427 ts_tablet_manager.cc:1939] T 759f70010ac74d198717bcbb5c1914c1 P bbdc71811c3b4416a687f18009ebc913: Deleting consensus metadata
I20260929 19:05:26.901115 23077 catalog_manager.cc:4841] TS bbdc71811c3b4416a687f18009ebc913 (127.8.158.66:38085): tablet 759f70010ac74d198717bcbb5c1914c1 (table test-table [id=80efa031b1ab4ca1ad03bd0e5456c21d]) successfully deleted
W20260929 19:05:26.968127 23424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:05:26.968321 23424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:05:26.968344 23424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:05:26.969475 23424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:05:26.969518 23424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:05:26.969544 23424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.67
I20260929 19:05:26.971035 23424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:34333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab
--principal=kudu/127.8.158.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.67:44703
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.158.67
--webserver_port=39981
--tserver_master_addrs=127.8.158.126:38373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.67
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:05:26.971299 23424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:05:26.971534 23424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:05:26.976400 23432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:26.976384 23435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:26.976416 23433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:05:26.976751 23424 server_base.cc:1034] running on GCE node
I20260929 19:05:26.976892 23424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:05:26.977073 23424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:05:26.978228 23424 hybrid_clock.cc:648] HybridClock initialized: now 1790708726978214 us; error 27 us; skew 500 ppm
Sep 29 19:05:26 dist-test-slave-nmsz krb5kdc[22515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708726, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:05:26.980768 23424 init.cc:377] Logged in from keytab as kudu/127.8.158.67@KRBTEST.COM (short username kudu)
I20260929 19:05:26.981117 23424 webserver.cc:460] Webserver started at http://127.8.158.67:39981/ using document root <none> and password file <none>
I20260929 19:05:26.981350 23424 fs_manager.cc:359] Metadata directory not provided
I20260929 19:05:26.981431 23424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:05:26.983062 23424 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 19:05:26.983788 23442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:26.983976 23424 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:05:26.984069 23424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-2/wal
uuid: "fd62ed66bc1040bbbe53171a1d114f7d"
format_stamp: "Formatted at 2026-09-29 19:05:26 on dist-test-slave-nmsz"
server_key: "0182c6a7cad9828c53d09fd544aec07d"
server_key_iv: "45fac18d108d03d341681772878180ff"
server_key_version: "encryptionkey@0"
I20260929 19:05:26.984424 23424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:05:26.998584 23424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:05:27.000597 23424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:05:27.000766 23424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:05:27.000983 23424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:05:27.001268 23424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:05:27.001328 23424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:27.001367 23424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:05:27.001395 23424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:27.007269 23424 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.67:44703
I20260929 19:05:27.007318 23555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.67:44703 every 8 connection(s)
I20260929 19:05:27.007622 23424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
I20260929 19:05:27.010254 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 23424
Sep 29 19:05:27 dist-test-slave-nmsz krb5kdc[22515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708726, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.67@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:05:27.017160 23560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:27.010641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57762 (local address 127.8.158.126:38373)
0929 19:05:27.010748 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:27.010750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:27.010772 (+ 22us) server_negotiation.cc:407] Connection header received
0929 19:05:27.010858 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:27.010859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:27.010872 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:27.010931 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:27.011595 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:27.012027 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:27.012988 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:27.013170 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:27.013219 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:27.014303 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:27.014305 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:27.014305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:27.014339 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:27.016019 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:27.016374 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:27.016375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:27.016376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:27.016438 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:27.016644 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:27.016645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:27.016646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:27.016756 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:27.016922 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:27.016961 (+ 39us) server_negotiation.cc:299] Negotiation successful
0929 19:05:27.017038 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":71,"threads_started":1}
I20260929 19:05:27.017540 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:27.008318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:39697 (local address 127.8.158.126:38373)
0929 19:05:27.008390 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:27.008392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:27.008600 (+ 208us) server_negotiation.cc:407] Connection header received
0929 19:05:27.009088 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:27.009089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:27.009102 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:27.009164 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:27.010080 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:27.010792 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:27.012089 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:27.012306 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:27.012352 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:27.014612 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:27.014614 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:27.014614 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:27.014621 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:27.016151 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:27.016607 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:27.016608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:27.016609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:27.016670 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:27.016914 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:27.016915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:27.016915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:27.017024 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:27.017098 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:27.017385 (+ 287us) server_negotiation.cc:299] Negotiation successful
0929 19:05:27.017460 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 19:05:27.017779 23558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:27.008325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.67:39697)
0929 19:05:27.008481 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:27.008512 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:27.008981 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:27.009236 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:27.009239 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:27.009793 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:05:27.009966 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:27.009970 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:27.010906 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:27.010907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:27.012001 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:27.012004 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:27.012381 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:27.012382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:27.012517 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:27.013002 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:05:27.013011 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:05:27.014522 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:05:27.016292 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:27.016299 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:27.016302 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:27.016527 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:27.016742 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:27.016743 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:27.016743 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:27.016843 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:27.017092 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:05:27.017098 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:05:27.017246 (+ 148us) client_negotiation.cc:769] Sending connection context
0929 19:05:27.017366 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 19:05:27.017522 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":85,"threads_started":1}
I20260929 19:05:27.018203 23556 heartbeater.cc:347] Connected to a master server at 127.8.158.126:38373
I20260929 19:05:27.018297 23556 heartbeater.cc:464] Registering TS with master...
I20260929 19:05:27.018481 23556 heartbeater.cc:511] Master 127.8.158.126:38373 requested a full tablet report, sending...
I20260929 19:05:27.018812 23093 ts_manager.cc:194] Registered new tserver with Master: fd62ed66bc1040bbbe53171a1d114f7d (127.8.158.67:44703)
I20260929 19:05:27.019171 23093 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.67@KRBTEST.COM'} at 127.8.158.67:39697
I20260929 19:05:27.028592 8825 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:05:27.031397 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:27.028949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57776 (local address 127.8.158.126:38373)
0929 19:05:27.028980 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:27.028982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:27.029078 (+ 96us) server_negotiation.cc:407] Connection header received
0929 19:05:27.029140 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:27.029141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:27.029167 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:27.029221 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:05:27.029314 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:27.029797 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:27.030617 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:27.030797 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:27.030853 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:27.031055 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:05:27.031177 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:05:27.031249 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:27.031289 (+ 40us) server_negotiation.cc:299] Negotiation successful
0929 19:05:27.031306 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260929 19:05:27.031625 23093 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:57776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:05:27.033074 23093 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:05:27.035223 23354 tablet_service.cc:1467] Processing CreateTablet for tablet a07def7d19c64fefa26370cb3b004ee2 (DEFAULT_TABLE table=test-table [id=746c055da12d4cccbf3a80d25c21c107]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:05:27.035429 23354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a07def7d19c64fefa26370cb3b004ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:05:27.037055 23312 tablet_bootstrap.cc:492] T a07def7d19c64fefa26370cb3b004ee2 P bbdc71811c3b4416a687f18009ebc913: Bootstrap starting.
I20260929 19:05:27.037168 23214 tablet_service.cc:1467] Processing CreateTablet for tablet a07def7d19c64fefa26370cb3b004ee2 (DEFAULT_TABLE table=test-table [id=746c055da12d4cccbf3a80d25c21c107]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:05:27.037345 23214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a07def7d19c64fefa26370cb3b004ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:05:27.037366 23312 tablet_bootstrap.cc:654] T a07def7d19c64fefa26370cb3b004ee2 P bbdc71811c3b4416a687f18009ebc913: Neither blocks nor log segments found. Creating new log.
I20260929 19:05:27.038143 23312 tablet_bootstrap.cc:492] T a07def7d19c64fefa26370cb3b004ee2 P bbdc71811c3b4416a687f18009ebc913: No bootstrap required, opened a new log
I20260929 19:05:27.038195 23312 ts_tablet_manager.cc:1397] T a07def7d19c64fefa26370cb3b004ee2 P bbdc71811c3b4416a687f18009ebc913: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 19:05:27.038367 23312 raft_consensus.cc:348] T a07def7d19c64fefa26370cb3b004ee2 P bbdc71811c3b4416a687f18009ebc913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd62ed66bc1040bbbe53171a1d114f7d" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 44703 } } peers { permanent_uuid: "bbdc71811c3b4416a687f18009ebc913" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38085 } } peers { permanent_uuid: "ffeec491014e4d6b89512c09a168dc9b" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 46257 } }
I20260929 19:05:27.038446 23312 raft_consensus.cc:374] T a07def7d19c64fefa26370cb3b004ee2 P bbdc71811c3b4416a687f18009ebc913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:05:27.038496 23312 raft_consensus.cc:729] T a07def7d19c64fefa26370cb3b004ee2 P bbdc71811c3b4416a687f18009ebc913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbdc71811c3b4416a687f18009ebc913, State: Initialized, Role: FOLLOWER
I20260929 19:05:27.038543 23312 consensus_queue.cc:260] T a07def7d19c64fefa26370cb3b004ee2 P bbdc71811c3b4416a687f18009ebc913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd62ed66bc1040bbbe53171a1d114f7d" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 44703 } } peers { permanent_uuid: "bbdc71811c3b4416a687f18009ebc913" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38085 } } peers { permanent_uuid: "ffeec491014e4d6b89512c09a168dc9b" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 46257 } }
I20260929 19:05:27.038642 23312 ts_tablet_manager.cc:1428] T a07def7d19c64fefa26370cb3b004ee2 P bbdc71811c3b4416a687f18009ebc913: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:27.039024 23173 tablet_bootstrap.cc:492] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b: Bootstrap starting.
I20260929 19:05:27.039319 23173 tablet_bootstrap.cc:654] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b: Neither blocks nor log segments found. Creating new log.
I20260929 19:05:27.039326 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:27.034983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.67:44703 (local address 127.0.0.1:49916)
0929 19:05:27.035030 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:27.035041 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:27.035080 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:27.035430 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:27.035431 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:27.035433 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:05:27.035480 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:27.035482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:27.036548 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:27.036549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:27.037677 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:27.037680 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:27.039086 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:27.039087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:27.039150 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:27.039154 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:05:27.039185 (+ 31us) client_negotiation.cc:240] Negotiation successful
0929 19:05:27.039239 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 19:05:27.039645 23562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:27.035105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49916 (local address 127.8.158.67:44703)
0929 19:05:27.035268 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:27.035272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:27.035282 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:05:27.035313 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:27.035315 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:27.035333 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:27.035426 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:05:27.035549 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:27.036435 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:27.037770 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:27.038983 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:27.039455 (+ 472us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:27.039526 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:27.039552 (+ 26us) server_negotiation.cc:299] Negotiation successful
0929 19:05:27.039585 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":113,"threads_started":1}
I20260929 19:05:27.039980 23173 tablet_bootstrap.cc:492] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b: No bootstrap required, opened a new log
I20260929 19:05:27.040020 23173 ts_tablet_manager.cc:1397] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260929 19:05:27.040122 23173 raft_consensus.cc:348] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd62ed66bc1040bbbe53171a1d114f7d" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 44703 } } peers { permanent_uuid: "bbdc71811c3b4416a687f18009ebc913" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38085 } } peers { permanent_uuid: "ffeec491014e4d6b89512c09a168dc9b" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 46257 } }
I20260929 19:05:27.040190 23173 raft_consensus.cc:374] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:05:27.040251 23173 raft_consensus.cc:729] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffeec491014e4d6b89512c09a168dc9b, State: Initialized, Role: FOLLOWER
I20260929 19:05:27.040308 23173 consensus_queue.cc:260] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd62ed66bc1040bbbe53171a1d114f7d" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 44703 } } peers { permanent_uuid: "bbdc71811c3b4416a687f18009ebc913" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38085 } } peers { permanent_uuid: "ffeec491014e4d6b89512c09a168dc9b" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 46257 } }
I20260929 19:05:27.040403 23173 ts_tablet_manager.cc:1428] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:27.040342 23490 tablet_service.cc:1467] Processing CreateTablet for tablet a07def7d19c64fefa26370cb3b004ee2 (DEFAULT_TABLE table=test-table [id=746c055da12d4cccbf3a80d25c21c107]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:05:27.040674 23490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a07def7d19c64fefa26370cb3b004ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:05:27.043483 23565 tablet_bootstrap.cc:492] T a07def7d19c64fefa26370cb3b004ee2 P fd62ed66bc1040bbbe53171a1d114f7d: Bootstrap starting.
I20260929 19:05:27.044023 23565 tablet_bootstrap.cc:654] T a07def7d19c64fefa26370cb3b004ee2 P fd62ed66bc1040bbbe53171a1d114f7d: Neither blocks nor log segments found. Creating new log.
I20260929 19:05:27.044498 23565 log.cc:824] T a07def7d19c64fefa26370cb3b004ee2 P fd62ed66bc1040bbbe53171a1d114f7d: Log is configured to *not* fsync() on all Append() calls
I20260929 19:05:27.045233 23565 tablet_bootstrap.cc:492] T a07def7d19c64fefa26370cb3b004ee2 P fd62ed66bc1040bbbe53171a1d114f7d: No bootstrap required, opened a new log
I20260929 19:05:27.045312 23565 ts_tablet_manager.cc:1397] T a07def7d19c64fefa26370cb3b004ee2 P fd62ed66bc1040bbbe53171a1d114f7d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:05:27.046725 23565 raft_consensus.cc:348] T a07def7d19c64fefa26370cb3b004ee2 P fd62ed66bc1040bbbe53171a1d114f7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd62ed66bc1040bbbe53171a1d114f7d" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 44703 } } peers { permanent_uuid: "bbdc71811c3b4416a687f18009ebc913" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38085 } } peers { permanent_uuid: "ffeec491014e4d6b89512c09a168dc9b" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 46257 } }
I20260929 19:05:27.046855 23565 raft_consensus.cc:374] T a07def7d19c64fefa26370cb3b004ee2 P fd62ed66bc1040bbbe53171a1d114f7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:05:27.046877 23565 raft_consensus.cc:729] T a07def7d19c64fefa26370cb3b004ee2 P fd62ed66bc1040bbbe53171a1d114f7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd62ed66bc1040bbbe53171a1d114f7d, State: Initialized, Role: FOLLOWER
I20260929 19:05:27.047001 23565 consensus_queue.cc:260] T a07def7d19c64fefa26370cb3b004ee2 P fd62ed66bc1040bbbe53171a1d114f7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd62ed66bc1040bbbe53171a1d114f7d" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 44703 } } peers { permanent_uuid: "bbdc71811c3b4416a687f18009ebc913" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38085 } } peers { permanent_uuid: "ffeec491014e4d6b89512c09a168dc9b" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 46257 } }
I20260929 19:05:27.047256 23565 ts_tablet_manager.cc:1428] T a07def7d19c64fefa26370cb3b004ee2 P fd62ed66bc1040bbbe53171a1d114f7d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:05:27.047335 23556 heartbeater.cc:503] Master 127.8.158.126:38373 was elected leader, sending a full tablet report...
W20260929 19:05:27.127641 23421 tablet.cc:2369] T a07def7d19c64fefa26370cb3b004ee2 P bbdc71811c3b4416a687f18009ebc913: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:05:27.209101 23567 raft_consensus.cc:482] T a07def7d19c64fefa26370cb3b004ee2 P fd62ed66bc1040bbbe53171a1d114f7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:05:27.209220 23567 raft_consensus.cc:504] T a07def7d19c64fefa26370cb3b004ee2 P fd62ed66bc1040bbbe53171a1d114f7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd62ed66bc1040bbbe53171a1d114f7d" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 44703 } } peers { permanent_uuid: "bbdc71811c3b4416a687f18009ebc913" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38085 } } peers { permanent_uuid: "ffeec491014e4d6b89512c09a168dc9b" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 46257 } }
I20260929 19:05:27.209555 23567 leader_election.cc:288] T a07def7d19c64fefa26370cb3b004ee2 P fd62ed66bc1040bbbe53171a1d114f7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbdc71811c3b4416a687f18009ebc913 (127.8.158.66:38085), ffeec491014e4d6b89512c09a168dc9b (127.8.158.65:46257)
I20260929 19:05:27.213617 23426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:27.209890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:54037 (local address 127.8.158.66:38085)
0929 19:05:27.209938 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:27.209940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:27.209949 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:05:27.209980 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:27.209981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:27.210005 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:27.210074 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:05:27.210351 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:27.211124 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:27.212621 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:27.213223 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:27.213287 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:27.213391 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:27.213503 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 19:05:27.213545 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 19:05:27.213614 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:27.209874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:34139 (local address 127.8.158.65:46257)
0929 19:05:27.209921 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:27.209923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:27.210126 (+ 203us) server_negotiation.cc:407] Connection header received
0929 19:05:27.210157 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:27.210157 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:27.210171 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:27.210213 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:05:27.210379 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:27.211124 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:27.212624 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:27.213189 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:27.213273 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:27.213284 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:27.213503 (+ 219us) server_negotiation.cc:299] Negotiation successful
0929 19:05:27.213545 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 19:05:27.213645 23558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:27.209774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.66:38085 (local address 127.8.158.67:54037)
0929 19:05:27.209841 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:27.209855 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:27.209914 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:27.210102 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:27.210103 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:27.210105 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:05:27.210173 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:27.210177 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:27.211232 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:27.211233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:27.212513 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:27.212514 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:27.213327 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:27.213328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:27.213413 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:27.213415 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 19:05:27.213477 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 19:05:27.213531 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":75}
I20260929 19:05:27.213651 23568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:27.209781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:46257 (local address 127.8.158.67:34139)
0929 19:05:27.210025 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:27.210033 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:27.210102 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:27.210237 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:27.210238 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:27.210240 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:05:27.210297 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:27.210299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:27.211232 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:27.211233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:27.212507 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:27.212510 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:27.213321 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:27.213322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:27.213408 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:27.213413 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 19:05:27.213477 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 19:05:27.213530 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":79,"thread_start_us":116,"threads_started":1}
I20260929 19:05:27.213989 23374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a07def7d19c64fefa26370cb3b004ee2" candidate_uuid: "fd62ed66bc1040bbbe53171a1d114f7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbdc71811c3b4416a687f18009ebc913" is_pre_election: true
I20260929 19:05:27.214010 23234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a07def7d19c64fefa26370cb3b004ee2" candidate_uuid: "fd62ed66bc1040bbbe53171a1d114f7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffeec491014e4d6b89512c09a168dc9b" is_pre_election: true
I20260929 19:05:27.214138 23374 raft_consensus.cc:2454] T a07def7d19c64fefa26370cb3b004ee2 P bbdc71811c3b4416a687f18009ebc913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd62ed66bc1040bbbe53171a1d114f7d in term 0.
I20260929 19:05:27.214185 23234 raft_consensus.cc:2454] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd62ed66bc1040bbbe53171a1d114f7d in term 0.
I20260929 19:05:27.214412 23446 leader_election.cc:302] T a07def7d19c64fefa26370cb3b004ee2 P fd62ed66bc1040bbbe53171a1d114f7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: fd62ed66bc1040bbbe53171a1d114f7d, ffeec491014e4d6b89512c09a168dc9b; no voters:
I20260929 19:05:27.214578 23567 raft_consensus.cc:2781] T a07def7d19c64fefa26370cb3b004ee2 P fd62ed66bc1040bbbe53171a1d114f7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:05:27.214672 23567 raft_consensus.cc:482] T a07def7d19c64fefa26370cb3b004ee2 P fd62ed66bc1040bbbe53171a1d114f7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:05:27.214709 23567 raft_consensus.cc:3037] T a07def7d19c64fefa26370cb3b004ee2 P fd62ed66bc1040bbbe53171a1d114f7d [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:05:27.215384 23567 raft_consensus.cc:504] T a07def7d19c64fefa26370cb3b004ee2 P fd62ed66bc1040bbbe53171a1d114f7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd62ed66bc1040bbbe53171a1d114f7d" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 44703 } } peers { permanent_uuid: "bbdc71811c3b4416a687f18009ebc913" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38085 } } peers { permanent_uuid: "ffeec491014e4d6b89512c09a168dc9b" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 46257 } }
I20260929 19:05:27.215533 23567 leader_election.cc:288] T a07def7d19c64fefa26370cb3b004ee2 P fd62ed66bc1040bbbe53171a1d114f7d [CANDIDATE]: Term 1 election: Requested vote from peers bbdc71811c3b4416a687f18009ebc913 (127.8.158.66:38085), ffeec491014e4d6b89512c09a168dc9b (127.8.158.65:46257)
I20260929 19:05:27.215694 23374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a07def7d19c64fefa26370cb3b004ee2" candidate_uuid: "fd62ed66bc1040bbbe53171a1d114f7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbdc71811c3b4416a687f18009ebc913"
I20260929 19:05:27.215690 23234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a07def7d19c64fefa26370cb3b004ee2" candidate_uuid: "fd62ed66bc1040bbbe53171a1d114f7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffeec491014e4d6b89512c09a168dc9b"
I20260929 19:05:27.215773 23374 raft_consensus.cc:3037] T a07def7d19c64fefa26370cb3b004ee2 P bbdc71811c3b4416a687f18009ebc913 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:05:27.215775 23234 raft_consensus.cc:3037] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:05:27.216658 23374 raft_consensus.cc:2454] T a07def7d19c64fefa26370cb3b004ee2 P bbdc71811c3b4416a687f18009ebc913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd62ed66bc1040bbbe53171a1d114f7d in term 1.
I20260929 19:05:27.216857 23443 leader_election.cc:302] T a07def7d19c64fefa26370cb3b004ee2 P fd62ed66bc1040bbbe53171a1d114f7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbdc71811c3b4416a687f18009ebc913, fd62ed66bc1040bbbe53171a1d114f7d; no voters:
I20260929 19:05:27.216941 23567 raft_consensus.cc:2781] T a07def7d19c64fefa26370cb3b004ee2 P fd62ed66bc1040bbbe53171a1d114f7d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:05:27.217068 23567 raft_consensus.cc:686] T a07def7d19c64fefa26370cb3b004ee2 P fd62ed66bc1040bbbe53171a1d114f7d [term 1 LEADER]: Becoming Leader. State: Replica: fd62ed66bc1040bbbe53171a1d114f7d, State: Running, Role: LEADER
I20260929 19:05:27.217094 23234 raft_consensus.cc:2454] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd62ed66bc1040bbbe53171a1d114f7d in term 1.
I20260929 19:05:27.217160 23567 consensus_queue.cc:237] T a07def7d19c64fefa26370cb3b004ee2 P fd62ed66bc1040bbbe53171a1d114f7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd62ed66bc1040bbbe53171a1d114f7d" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 44703 } } peers { permanent_uuid: "bbdc71811c3b4416a687f18009ebc913" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38085 } } peers { permanent_uuid: "ffeec491014e4d6b89512c09a168dc9b" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 46257 } }
I20260929 19:05:27.217931 23093 catalog_manager.cc:5495] T a07def7d19c64fefa26370cb3b004ee2 P fd62ed66bc1040bbbe53171a1d114f7d reported cstate change: term changed from 0 to 1, leader changed from <none> to fd62ed66bc1040bbbe53171a1d114f7d (127.8.158.67). New cstate: current_term: 1 leader_uuid: "fd62ed66bc1040bbbe53171a1d114f7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd62ed66bc1040bbbe53171a1d114f7d" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 44703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbdc71811c3b4416a687f18009ebc913" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ffeec491014e4d6b89512c09a168dc9b" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 46257 } health_report { overall_health: UNKNOWN } } }
W20260929 19:05:27.218761 23282 tablet.cc:2369] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 19:05:27.258263 23557 tablet.cc:2369] T a07def7d19c64fefa26370cb3b004ee2 P fd62ed66bc1040bbbe53171a1d114f7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:05:27.266224 23562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:27.264012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49932 (local address 127.8.158.67:44703)
0929 19:05:27.264080 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:27.264082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:27.264090 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:05:27.264131 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:27.264132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:27.264147 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:27.264205 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:05:27.264334 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:27.264784 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:27.265508 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:27.265699 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:27.265760 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:27.265904 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:05:27.265993 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:05:27.266050 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:27.266085 (+ 35us) server_negotiation.cc:299] Negotiation successful
0929 19:05:27.266101 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 19:05:27.267858 23234 raft_consensus.cc:1261] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b [term 1 FOLLOWER]: Refusing update from remote peer fd62ed66bc1040bbbe53171a1d114f7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:05:27.267867 23374 raft_consensus.cc:1261] T a07def7d19c64fefa26370cb3b004ee2 P bbdc71811c3b4416a687f18009ebc913 [term 1 FOLLOWER]: Refusing update from remote peer fd62ed66bc1040bbbe53171a1d114f7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:05:27.268200 23567 consensus_queue.cc:1035] T a07def7d19c64fefa26370cb3b004ee2 P fd62ed66bc1040bbbe53171a1d114f7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbdc71811c3b4416a687f18009ebc913" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:05:27.268330 23569 consensus_queue.cc:1035] T a07def7d19c64fefa26370cb3b004ee2 P fd62ed66bc1040bbbe53171a1d114f7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffeec491014e4d6b89512c09a168dc9b" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 46257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:05:27.270370 23573 mvcc.cc:205] Tried to move back new op lower bound from 7334742946886684672 to 7334742946682183680. Current Snapshot: MvccSnapshot[applied={T|T < 7334742946886684672}]
I20260929 19:05:27.272745 23093 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:57776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:05:27.272837 23093 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:57776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:05:27.273531 23093 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054: Sending DeleteTablet for 3 replicas of tablet a07def7d19c64fefa26370cb3b004ee2
I20260929 19:05:27.273871 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 23131
I20260929 19:05:27.273893 23214 tablet_service.cc:1514] Processing DeleteTablet for tablet a07def7d19c64fefa26370cb3b004ee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:05:27 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:34014
I20260929 19:05:27.273896 23354 tablet_service.cc:1514] Processing DeleteTablet for tablet a07def7d19c64fefa26370cb3b004ee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:05:27 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:38688
I20260929 19:05:27.274439 23427 tablet_replica.cc:330] T a07def7d19c64fefa26370cb3b004ee2 P bbdc71811c3b4416a687f18009ebc913: stopping tablet replica
I20260929 19:05:27.274497 23427 raft_consensus.cc:2229] T a07def7d19c64fefa26370cb3b004ee2 P bbdc71811c3b4416a687f18009ebc913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:05:27.274551 23427 raft_consensus.cc:2258] T a07def7d19c64fefa26370cb3b004ee2 P bbdc71811c3b4416a687f18009ebc913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:05:27.274880 23427 ts_tablet_manager.cc:1905] T a07def7d19c64fefa26370cb3b004ee2 P bbdc71811c3b4416a687f18009ebc913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:05:27.276366 23427 ts_tablet_manager.cc:1918] T a07def7d19c64fefa26370cb3b004ee2 P bbdc71811c3b4416a687f18009ebc913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:05:27.276376 23490 tablet_service.cc:1514] Processing DeleteTablet for tablet a07def7d19c64fefa26370cb3b004ee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:05:27 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:49916
I20260929 19:05:27.276423 23427 log.cc:1196] T a07def7d19c64fefa26370cb3b004ee2 P bbdc71811c3b4416a687f18009ebc913: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-1/wal/wals/a07def7d19c64fefa26370cb3b004ee2
I20260929 19:05:27.276644 23427 ts_tablet_manager.cc:1939] T a07def7d19c64fefa26370cb3b004ee2 P bbdc71811c3b4416a687f18009ebc913: Deleting consensus metadata
I20260929 19:05:27.276962 23077 catalog_manager.cc:4841] TS bbdc71811c3b4416a687f18009ebc913 (127.8.158.66:38085): tablet a07def7d19c64fefa26370cb3b004ee2 (table test-table [id=746c055da12d4cccbf3a80d25c21c107]) successfully deleted
I20260929 19:05:27.277069 23582 tablet_replica.cc:330] T a07def7d19c64fefa26370cb3b004ee2 P fd62ed66bc1040bbbe53171a1d114f7d: stopping tablet replica
I20260929 19:05:27.277176 23582 raft_consensus.cc:2229] T a07def7d19c64fefa26370cb3b004ee2 P fd62ed66bc1040bbbe53171a1d114f7d [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:05:27.277305 23582 raft_consensus.cc:2258] T a07def7d19c64fefa26370cb3b004ee2 P fd62ed66bc1040bbbe53171a1d114f7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:05:27.277649 23582 ts_tablet_manager.cc:1905] T a07def7d19c64fefa26370cb3b004ee2 P fd62ed66bc1040bbbe53171a1d114f7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:05:27.279567 23582 ts_tablet_manager.cc:1918] T a07def7d19c64fefa26370cb3b004ee2 P fd62ed66bc1040bbbe53171a1d114f7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:05:27.279640 23582 log.cc:1196] T a07def7d19c64fefa26370cb3b004ee2 P fd62ed66bc1040bbbe53171a1d114f7d: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-2/wal/wals/a07def7d19c64fefa26370cb3b004ee2
I20260929 19:05:27.279897 23582 ts_tablet_manager.cc:1939] T a07def7d19c64fefa26370cb3b004ee2 P fd62ed66bc1040bbbe53171a1d114f7d: Deleting consensus metadata
I20260929 19:05:27.280296 23079 catalog_manager.cc:4841] TS fd62ed66bc1040bbbe53171a1d114f7d (127.8.158.67:44703): tablet a07def7d19c64fefa26370cb3b004ee2 (table test-table [id=746c055da12d4cccbf3a80d25c21c107]) successfully deleted
W20260929 19:05:27.280493 23080 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.158.65:46257 (error 108)
I20260929 19:05:27.280884 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 23286
I20260929 19:05:27.280918 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:27.280806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:46257 (local address 127.0.0.1:34022)
0929 19:05:27.280845 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:27.280882 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.65:46257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260929 19:05:27.282644 23080 catalog_manager.cc:4584] TS ffeec491014e4d6b89512c09a168dc9b (127.8.158.65:46257): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a07def7d19c64fefa26370cb3b004ee2: Network error: Client connection negotiation failed: client connection to 127.8.158.65:46257: connect: Connection refused (error 111)
I20260929 19:05:27.286641 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 23424
I20260929 19:05:27.292130 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 23060
I20260929 19:05:27.297240 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:38373
--webserver_interface=127.8.158.126
--webserver_port=33551
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:34333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:38373
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:05:27.375209 23584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:05:27.375422 23584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:05:27.375458 23584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:05:27.376559 23584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:05:27.376626 23584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:05:27.376665 23584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:05:27.376714 23584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:05:27.376744 23584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:05:27.378147 23584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:34333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:38373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:38373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.158.126
--webserver_port=33551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:05:27.378413 23584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:05:27.378631 23584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:05:27.383262 23592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:27.383323 23590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:27.383345 23589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:05:27.383517 23584 server_base.cc:1034] running on GCE node
I20260929 19:05:27.383774 23584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:05:27.384063 23584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:05:27.385208 23584 hybrid_clock.cc:648] HybridClock initialized: now 1790708727385185 us; error 40 us; skew 500 ppm
Sep 29 19:05:27 dist-test-slave-nmsz krb5kdc[22515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708727, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:05:27.387758 23584 init.cc:377] Logged in from keytab as kudu/127.8.158.126@KRBTEST.COM (short username kudu)
I20260929 19:05:27.388156 23584 webserver.cc:460] Webserver started at http://127.8.158.126:33551/ using document root <none> and password file <none>
I20260929 19:05:27.388381 23584 fs_manager.cc:359] Metadata directory not provided
I20260929 19:05:27.388449 23584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:05:27.389707 23584 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:05:27.390455 23599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:27.390641 23584 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:05:27.390760 23584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "7181e789b0c24f22b65f8fdd3e231054"
format_stamp: "Formatted at 2026-09-29 19:05:25 on dist-test-slave-nmsz"
server_key: "327656b0ce57b1499e5606bdaeaa9ab5"
server_key_iv: "225f646728bcf3f8368a57734be38341"
server_key_version: "encryptionkey@0"
I20260929 19:05:27.391104 23584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:05:27.401189 23584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:05:27.403272 23584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:05:27.403438 23584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:05:27.407513 23584 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.126:38373
I20260929 19:05:27.407539 23651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.126:38373 every 8 connection(s)
I20260929 19:05:27.407915 23584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:05:27.409304 23652 sys_catalog.cc:263] Verifying existing consensus state
I20260929 19:05:27.409914 23652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054: Bootstrap starting.
I20260929 19:05:27.411609 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 23584
I20260929 19:05:27.411938 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.65:46257
--local_ip_for_outbound_sockets=127.8.158.65
--tserver_master_addrs=127.8.158.126:38373
--webserver_port=43355
--webserver_interface=127.8.158.65
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab
--principal=kudu/127.8.158.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:34333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:05:27.418452 23652 log.cc:824] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054: Log is configured to *not* fsync() on all Append() calls
I20260929 19:05:27.420454 23652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 19:05:27.420734 23652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054: Bootstrap complete.
I20260929 19:05:27.422931 23652 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7181e789b0c24f22b65f8fdd3e231054" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 38373 } }
I20260929 19:05:27.423157 23652 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7181e789b0c24f22b65f8fdd3e231054, State: Initialized, Role: FOLLOWER
I20260929 19:05:27.423349 23652 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7181e789b0c24f22b65f8fdd3e231054" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 38373 } }
I20260929 19:05:27.423437 23652 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:05:27.423488 23652 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:05:27.423538 23652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [term 2 FOLLOWER]: Advancing to term 3
I20260929 19:05:27.424428 23652 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7181e789b0c24f22b65f8fdd3e231054" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 38373 } }
I20260929 19:05:27.424541 23652 leader_election.cc:302] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7181e789b0c24f22b65f8fdd3e231054; no voters:
I20260929 19:05:27.424718 23652 leader_election.cc:288] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 19:05:27.424752 23656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 19:05:27.424861 23656 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [term 3 LEADER]: Becoming Leader. State: Replica: 7181e789b0c24f22b65f8fdd3e231054, State: Running, Role: LEADER
I20260929 19:05:27.424968 23652 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:05:27.424950 23656 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7181e789b0c24f22b65f8fdd3e231054" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 38373 } }
I20260929 19:05:27.425222 23656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7181e789b0c24f22b65f8fdd3e231054. Latest consensus state: current_term: 3 leader_uuid: "7181e789b0c24f22b65f8fdd3e231054" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7181e789b0c24f22b65f8fdd3e231054" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 38373 } } }
I20260929 19:05:27.425287 23656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [sys.catalog]: This master's current role is: LEADER
I20260929 19:05:27.425486 23657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "7181e789b0c24f22b65f8fdd3e231054" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7181e789b0c24f22b65f8fdd3e231054" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 38373 } } }
I20260929 19:05:27.425549 23657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7181e789b0c24f22b65f8fdd3e231054 [sys.catalog]: This master's current role is: LEADER
I20260929 19:05:27.426110 23660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:05:27.426862 23660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:05:27.426972 23660 catalog_manager.cc:1259] Loaded cluster ID: 1cb79e5cf82d40e78d3c5340b60f0ca3
I20260929 19:05:27.427219 23660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 19:05:27.427446 23660 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f29db1749ad google::LogMessage::Fail() at ??:0
@ 0x7f29db17788a google::LogMessage::SendToLog() at ??:0
@ 0x7f29db1744de google::LogMessage::Flush() at ??:0
@ 0x7f29db175a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f29e30950f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f29e309e8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f29dd0420d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f29dd037954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f29dde9b6db start_thread at ??:0
@ 0x7f29d9c7261f clone at ??:0
W20260929 19:05:27.502521 23654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:05:27.502693 23654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:05:27.502717 23654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:05:27.503995 23654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:05:27.504038 23654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:05:27.504065 23654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.65
I20260929 19:05:27.505486 23654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:34333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab
--principal=kudu/127.8.158.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.65:46257
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.158.65
--webserver_port=43355
--tserver_master_addrs=127.8.158.126:38373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.65
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:05:27.505700 23654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:05:27.505929 23654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:05:27.510747 23688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:27.510842 23689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:27.510739 23691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:05:27.511197 23654 server_base.cc:1034] running on GCE node
I20260929 19:05:27.511382 23654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:05:27.511632 23654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:05:27.512797 23654 hybrid_clock.cc:648] HybridClock initialized: now 1790708727512777 us; error 34 us; skew 500 ppm
Sep 29 19:05:27 dist-test-slave-nmsz krb5kdc[22515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708727, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:05:27.515547 23654 init.cc:377] Logged in from keytab as kudu/127.8.158.65@KRBTEST.COM (short username kudu)
I20260929 19:05:27.515990 23654 webserver.cc:460] Webserver started at http://127.8.158.65:43355/ using document root <none> and password file <none>
I20260929 19:05:27.516220 23654 fs_manager.cc:359] Metadata directory not provided
I20260929 19:05:27.516286 23654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:05:27.517652 23654 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:05:27.518388 23698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:27.518599 23654 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:05:27.518683 23654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-0/wal
uuid: "ffeec491014e4d6b89512c09a168dc9b"
format_stamp: "Formatted at 2026-09-29 19:05:25 on dist-test-slave-nmsz"
server_key: "c303953b3fd52bb2c7c754497c42d7fa"
server_key_iv: "bc0a2b9477c4cca25bed3ec569d2e990"
server_key_version: "encryptionkey@0"
I20260929 19:05:27.519014 23654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:05:27.525483 23654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:05:27.527722 23654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:05:27.527869 23654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:05:27.528092 23654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:05:27.528488 23705 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 19:05:27.530486 23654 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 19:05:27.530546 23654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260929 19:05:27.530587 23654 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 19:05:27.531155 23654 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 19:05:27.531210 23654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260929 19:05:27.531286 23705 tablet_bootstrap.cc:492] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b: Bootstrap starting.
I20260929 19:05:27.538170 23654 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.65:46257
I20260929 19:05:27.538273 23812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.65:46257 every 8 connection(s)
I20260929 19:05:27.538570 23654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
I20260929 19:05:27.544456 23705 log.cc:824] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b: Log is configured to *not* fsync() on all Append() calls
I20260929 19:05:27.547057 23705 tablet_bootstrap.cc:492] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 19:05:27.547139 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 23654
I20260929 19:05:27.547358 23705 tablet_bootstrap.cc:492] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b: Bootstrap complete.
I20260929 19:05:27.547420 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.66:38085
--local_ip_for_outbound_sockets=127.8.158.66
--tserver_master_addrs=127.8.158.126:38373
--webserver_port=32849
--webserver_interface=127.8.158.66
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab
--principal=kudu/127.8.158.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:34333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:05:27.548524 23705 ts_tablet_manager.cc:1397] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.004s
I20260929 19:05:27.549978 23705 raft_consensus.cc:348] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd62ed66bc1040bbbe53171a1d114f7d" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 44703 } } peers { permanent_uuid: "bbdc71811c3b4416a687f18009ebc913" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38085 } } peers { permanent_uuid: "ffeec491014e4d6b89512c09a168dc9b" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 46257 } }
I20260929 19:05:27.550220 23705 raft_consensus.cc:729] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffeec491014e4d6b89512c09a168dc9b, State: Initialized, Role: FOLLOWER
I20260929 19:05:27.550385 23705 consensus_queue.cc:260] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd62ed66bc1040bbbe53171a1d114f7d" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 44703 } } peers { permanent_uuid: "bbdc71811c3b4416a687f18009ebc913" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38085 } } peers { permanent_uuid: "ffeec491014e4d6b89512c09a168dc9b" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 46257 } }
I20260929 19:05:27.550657 23705 ts_tablet_manager.cc:1428] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
W20260929 19:05:27.654408 23817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:05:27.654589 23817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:05:27.654613 23817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:05:27.655762 23817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:05:27.655802 23817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:05:27.655828 23817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.66
I20260929 19:05:27.657233 23817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:34333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab
--principal=kudu/127.8.158.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.66:38085
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.158.66
--webserver_port=32849
--tserver_master_addrs=127.8.158.126:38373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.66
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:05:27.657454 23817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:05:27.657689 23817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:05:27.663012 23824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:27.663026 23823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:27.663020 23826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:05:27.663357 23817 server_base.cc:1034] running on GCE node
I20260929 19:05:27.663548 23817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:05:27.663807 23817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:05:27.664963 23817 hybrid_clock.cc:648] HybridClock initialized: now 1790708727664947 us; error 30 us; skew 500 ppm
Sep 29 19:05:27 dist-test-slave-nmsz krb5kdc[22515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708727, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:05:27.667605 23817 init.cc:377] Logged in from keytab as kudu/127.8.158.66@KRBTEST.COM (short username kudu)
I20260929 19:05:27.668049 23817 webserver.cc:460] Webserver started at http://127.8.158.66:32849/ using document root <none> and password file <none>
I20260929 19:05:27.668265 23817 fs_manager.cc:359] Metadata directory not provided
I20260929 19:05:27.668331 23817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:05:27.669682 23817 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:05:27.670305 23833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:27.670471 23817 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:05:27.670565 23817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-1/wal
uuid: "bbdc71811c3b4416a687f18009ebc913"
format_stamp: "Formatted at 2026-09-29 19:05:26 on dist-test-slave-nmsz"
server_key: "860585501b14d580da7a42459a7852e0"
server_key_iv: "1687d7ca24be53ddd9e1f1c6b371f095"
server_key_version: "encryptionkey@0"
I20260929 19:05:27.670893 23817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260929 19:05:27.676093 23815 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:05:27.539214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.65:50039)
0929 19:05:27.539381 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:27.539420 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:27.539813 (+ 393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:27.675888 (+136075us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":103,"threads_started":1}
W20260929 19:05:27.676363 23813 heartbeater.cc:644] Failed to heartbeat to 127.8.158.126:38373 (0 consecutive failures): Network error: Failed to ping master at 127.8.158.126:38373: Client connection negotiation failed: client connection to 127.8.158.126:38373: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 19:05:27.676649 23815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:27.676547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.65:51855)
0929 19:05:27.676593 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:27.676630 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260929 19:05:27.676970 23815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:27.676897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.65:41465)
0929 19:05:27.676933 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:27.676955 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260929 19:05:27.677083 23813 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 19:05:27.687120 23817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:05:27.688900 23817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:05:27.689033 23817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:05:27.689256 23817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:05:27.689522 23817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:05:27.689553 23817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:27.689574 23817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:05:27.689618 23817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:27.695305 23817 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.66:38085
I20260929 19:05:27.695343 23946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.66:38085 every 8 connection(s)
I20260929 19:05:27.695699 23817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
I20260929 19:05:27.696641 23949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:27.696299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.66:43653)
0929 19:05:27.696444 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:27.696511 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":71,"threads_started":1}
W20260929 19:05:27.697026 23947 heartbeater.cc:644] Failed to heartbeat to 127.8.158.126:38373 (0 consecutive failures): Network error: Failed to ping master at 127.8.158.126:38373: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
I20260929 19:05:27.697285 23949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:27.697189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.66:59217)
0929 19:05:27.697241 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:27.697267 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 19:05:27.697506 23949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:27.697454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.66:54627)
0929 19:05:27.697475 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:27.697493 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260929 19:05:27.697608 23947 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 19:05:27.703856 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 23817
I20260929 19:05:27.704208 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.67:44703
--local_ip_for_outbound_sockets=127.8.158.67
--tserver_master_addrs=127.8.158.126:38373
--webserver_port=39981
--webserver_interface=127.8.158.67
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab
--principal=kudu/127.8.158.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:34333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:05:27.784049 23950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:05:27.784261 23950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:05:27.784300 23950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:05:27.785460 23950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:05:27.785524 23950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:05:27.785570 23950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.67
I20260929 19:05:27.787019 23950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:34333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab
--principal=kudu/127.8.158.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.67:44703
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.158.67
--webserver_port=39981
--tserver_master_addrs=127.8.158.126:38373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.67
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:05:27.787264 23950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:05:27.787484 23950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:05:27.792219 23958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:27.792233 23955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:05:27.792323 23950 server_base.cc:1034] running on GCE node
W20260929 19:05:27.792472 23956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:05:27.792739 23950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:05:27.792990 23950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:05:27.794160 23950 hybrid_clock.cc:648] HybridClock initialized: now 1790708727794137 us; error 35 us; skew 500 ppm
Sep 29 19:05:27 dist-test-slave-nmsz krb5kdc[22515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708727, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:05:27.796574 23950 init.cc:377] Logged in from keytab as kudu/127.8.158.67@KRBTEST.COM (short username kudu)
I20260929 19:05:27.797024 23950 webserver.cc:460] Webserver started at http://127.8.158.67:39981/ using document root <none> and password file <none>
I20260929 19:05:27.797240 23950 fs_manager.cc:359] Metadata directory not provided
I20260929 19:05:27.797286 23950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:05:27.798561 23950 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:05:27.799119 23965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:27.799328 23950 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:05:27.799393 23950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-2/wal
uuid: "fd62ed66bc1040bbbe53171a1d114f7d"
format_stamp: "Formatted at 2026-09-29 19:05:26 on dist-test-slave-nmsz"
server_key: "0182c6a7cad9828c53d09fd544aec07d"
server_key_iv: "45fac18d108d03d341681772878180ff"
server_key_version: "encryptionkey@0"
I20260929 19:05:27.799713 23950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:05:27.812031 23950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:05:27.813848 23950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:05:27.813987 23950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:05:27.814208 23950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:05:27.814538 23950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:05:27.814571 23950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:27.814591 23950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:05:27.814634 23950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:27.820021 23950 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.67:44703
I20260929 19:05:27.820093 24078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.67:44703 every 8 connection(s)
I20260929 19:05:27.820395 23950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
I20260929 19:05:27.821350 24081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:27.820996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.67:39707)
0929 19:05:27.821144 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:27.821204 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":71,"threads_started":1}
W20260929 19:05:27.821770 24079 heartbeater.cc:644] Failed to heartbeat to 127.8.158.126:38373 (0 consecutive failures): Network error: Failed to ping master at 127.8.158.126:38373: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
I20260929 19:05:27.822060 24081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:27.821975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.67:37053)
0929 19:05:27.822020 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:27.822045 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 19:05:27.822306 24081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:27.822259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.67:59517)
0929 19:05:27.822278 (+ 19us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:27.822295 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260929 19:05:27.822446 24079 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 19:05:27.828768 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 23950
I20260929 19:05:28.677659 24083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:28.677447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.65:37271)
0929 19:05:28.677557 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:28.677595 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":55,"threads_started":1}
I20260929 19:05:28.698113 24084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:28.697910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.66:59489)
0929 19:05:28.698042 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:28.698069 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
I20260929 19:05:28.822899 24085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:28.822709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.67:59597)
0929 19:05:28.822827 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:28.822856 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":72,"threads_started":1}
I20260929 19:05:29.118171 24086 raft_consensus.cc:482] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:05:29.118294 24086 raft_consensus.cc:504] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd62ed66bc1040bbbe53171a1d114f7d" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 44703 } } peers { permanent_uuid: "bbdc71811c3b4416a687f18009ebc913" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38085 } } peers { permanent_uuid: "ffeec491014e4d6b89512c09a168dc9b" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 46257 } }
I20260929 19:05:29.118714 24086 leader_election.cc:288] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fd62ed66bc1040bbbe53171a1d114f7d (127.8.158.67:44703), bbdc71811c3b4416a687f18009ebc913 (127.8.158.66:38085)
Sep 29 19:05:29 dist-test-slave-nmsz krb5kdc[22515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708727, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for kudu/127.8.158.66@KRBTEST.COM
Sep 29 19:05:29 dist-test-slave-nmsz krb5kdc[22515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708727, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for kudu/127.8.158.67@KRBTEST.COM
I20260929 19:05:29.129046 24083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:29.118876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.67:44703 (local address 127.8.158.65:55157)
0929 19:05:29.118948 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:29.118959 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:29.119013 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:29.119842 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:29.119845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:29.120233 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:05:29.120402 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:29.120407 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:29.121533 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:29.121534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:29.122475 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:29.122477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:29.123034 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:29.123035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:29.123143 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:29.123788 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:05:29.123794 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:05:29.125780 (+ 1986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:05:29.127579 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:29.127580 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:29.127581 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:29.127805 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:29.128067 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:29.128068 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:29.128068 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:29.128150 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:29.128516 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:05:29.128522 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:05:29.128654 (+ 132us) client_negotiation.cc:769] Sending connection context
0929 19:05:29.128760 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 19:05:29.128945 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":2}
I20260929 19:05:29.129060 24089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:29.118994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:55157 (local address 127.8.158.67:44703)
0929 19:05:29.119153 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:29.119171 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:29.119195 (+ 24us) server_negotiation.cc:407] Connection header received
0929 19:05:29.119614 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:29.119616 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:29.119657 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:29.119798 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:29.120517 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:29.121443 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:29.122557 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:29.122945 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:29.123055 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:29.125915 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:29.125918 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:29.125919 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:29.125932 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:29.127475 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:29.127898 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:29.127900 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:29.127901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:29.127987 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:29.128224 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:29.128225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:29.128225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:29.128441 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:29.128535 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:29.128835 (+ 300us) server_negotiation.cc:299] Negotiation successful
0929 19:05:29.128954 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":53,"threads_started":1}
I20260929 19:05:29.129070 24088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:29.118989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:52995 (local address 127.8.158.66:38085)
0929 19:05:29.119145 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:29.119163 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:29.119191 (+ 28us) server_negotiation.cc:407] Connection header received
0929 19:05:29.119625 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:29.119627 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:29.119663 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:29.119817 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:29.120518 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:29.121505 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:29.122562 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:29.122950 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:29.123070 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:29.125422 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:29.125426 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:29.125426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:29.125436 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:29.127389 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:29.127869 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:29.127871 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:29.127872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:29.127963 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:29.128214 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:29.128215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:29.128215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:29.128441 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:29.128534 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:29.128815 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 19:05:29.128940 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":55,"threads_started":1}
I20260929 19:05:29.129161 24087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:29.118894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.66:38085 (local address 127.8.158.65:52995)
0929 19:05:29.119040 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:29.119049 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:29.119087 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:29.119844 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:29.119846 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:29.120232 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:05:29.120402 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:29.120407 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:29.121593 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:29.121593 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:29.122477 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:29.122479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:29.123050 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:29.123051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:29.123142 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:29.123732 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:05:29.123740 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:05:29.125284 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:05:29.127497 (+ 2213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:29.127504 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:29.127506 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:29.127773 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:29.128035 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:29.128036 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:29.128036 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:29.128146 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:29.128516 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:05:29.128522 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:05:29.128653 (+ 131us) client_negotiation.cc:769] Sending connection context
0929 19:05:29.128760 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 19:05:29.129028 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":186,"thread_start_us":85,"threads_started":1}
W20260929 19:05:29.129846 23699 leader_election.cc:341] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bbdc71811c3b4416a687f18009ebc913 (127.8.158.66:38085): Not found: Tablet not found: a07def7d19c64fefa26370cb3b004ee2
W20260929 19:05:29.130460 23701 leader_election.cc:341] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fd62ed66bc1040bbbe53171a1d114f7d (127.8.158.67:44703): Not found: Tablet not found: a07def7d19c64fefa26370cb3b004ee2
I20260929 19:05:29.130515 23701 leader_election.cc:302] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ffeec491014e4d6b89512c09a168dc9b; no voters: bbdc71811c3b4416a687f18009ebc913, fd62ed66bc1040bbbe53171a1d114f7d
I20260929 19:05:29.130620 24086 raft_consensus.cc:2726] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 19:05:29.678301 24090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:29.678103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.65:51991)
0929 19:05:29.678217 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:29.678258 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":61,"threads_started":1}
I20260929 19:05:29.698699 24091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:29.698521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.66:45569)
0929 19:05:29.698634 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:29.698666 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":47,"threads_started":1}
I20260929 19:05:29.823478 24092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:29.823276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.67:53979)
0929 19:05:29.823401 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:29.823434 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":62,"threads_started":1}
I20260929 19:05:30.678996 24093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:30.678803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.65:40369)
0929 19:05:30.678917 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:30.678951 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260929 19:05:30.699328 24094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:30.699153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.66:57717)
0929 19:05:30.699262 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:30.699294 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":53,"threads_started":1}
I20260929 19:05:30.824147 24095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:30.823925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.67:41609)
0929 19:05:30.824074 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:30.824109 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":57,"threads_started":1}
I20260929 19:05:31.059906 24096 raft_consensus.cc:482] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:05:31.060025 24096 raft_consensus.cc:504] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd62ed66bc1040bbbe53171a1d114f7d" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 44703 } } peers { permanent_uuid: "bbdc71811c3b4416a687f18009ebc913" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38085 } } peers { permanent_uuid: "ffeec491014e4d6b89512c09a168dc9b" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 46257 } }
I20260929 19:05:31.060240 24096 leader_election.cc:288] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fd62ed66bc1040bbbe53171a1d114f7d (127.8.158.67:44703), bbdc71811c3b4416a687f18009ebc913 (127.8.158.66:38085)
W20260929 19:05:31.060721 23699 leader_election.cc:341] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bbdc71811c3b4416a687f18009ebc913 (127.8.158.66:38085): Not found: Tablet not found: a07def7d19c64fefa26370cb3b004ee2
W20260929 19:05:31.060787 23701 leader_election.cc:341] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fd62ed66bc1040bbbe53171a1d114f7d (127.8.158.67:44703): Not found: Tablet not found: a07def7d19c64fefa26370cb3b004ee2
I20260929 19:05:31.060814 23699 leader_election.cc:302] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ffeec491014e4d6b89512c09a168dc9b; no voters: bbdc71811c3b4416a687f18009ebc913, fd62ed66bc1040bbbe53171a1d114f7d
I20260929 19:05:31.060909 24096 raft_consensus.cc:2726] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 19:05:31.679740 24097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:31.679510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.65:58237)
0929 19:05:31.679649 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:31.679695 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":62,"threads_started":1}
I20260929 19:05:31.699965 24098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:31.699773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.66:45497)
0929 19:05:31.699889 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:31.699926 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":67,"threads_started":1}
I20260929 19:05:31.824895 24099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:31.824685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.67:58763)
0929 19:05:31.824824 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:31.824857 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":67,"threads_started":1}
I20260929 19:05:32.680677 24100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:32.680444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.65:47771)
0929 19:05:32.680584 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:32.680623 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":63,"threads_started":1}
I20260929 19:05:32.700562 24101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:32.700383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.66:42103)
0929 19:05:32.700494 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:32.700527 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":55,"threads_started":1}
I20260929 19:05:32.825578 24102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:32.825366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.67:44849)
0929 19:05:32.825499 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:32.825534 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":71,"threads_started":1}
I20260929 19:05:33.471621 24103 raft_consensus.cc:482] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:05:33.471709 24103 raft_consensus.cc:504] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd62ed66bc1040bbbe53171a1d114f7d" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 44703 } } peers { permanent_uuid: "bbdc71811c3b4416a687f18009ebc913" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38085 } } peers { permanent_uuid: "ffeec491014e4d6b89512c09a168dc9b" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 46257 } }
I20260929 19:05:33.471922 24103 leader_election.cc:288] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fd62ed66bc1040bbbe53171a1d114f7d (127.8.158.67:44703), bbdc71811c3b4416a687f18009ebc913 (127.8.158.66:38085)
W20260929 19:05:33.472411 23699 leader_election.cc:341] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bbdc71811c3b4416a687f18009ebc913 (127.8.158.66:38085): Not found: Tablet not found: a07def7d19c64fefa26370cb3b004ee2
W20260929 19:05:33.472491 23701 leader_election.cc:341] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fd62ed66bc1040bbbe53171a1d114f7d (127.8.158.67:44703): Not found: Tablet not found: a07def7d19c64fefa26370cb3b004ee2
I20260929 19:05:33.472525 23701 leader_election.cc:302] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ffeec491014e4d6b89512c09a168dc9b; no voters: bbdc71811c3b4416a687f18009ebc913, fd62ed66bc1040bbbe53171a1d114f7d
I20260929 19:05:33.472613 24103 raft_consensus.cc:2726] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 19:05:33.681445 24104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:33.681205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.65:41841)
0929 19:05:33.681350 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:33.681387 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":62,"threads_started":1}
I20260929 19:05:33.701330 24105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:33.701142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.66:57087)
0929 19:05:33.701266 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:33.701294 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":74,"threads_started":1}
I20260929 19:05:33.826304 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:33.826121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.67:52771)
0929 19:05:33.826235 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:33.826266 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":60,"threads_started":1}
I20260929 19:05:34.682138 24107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:34.681924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.65:48891)
0929 19:05:34.682048 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:34.682078 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
I20260929 19:05:34.701876 24108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:34.701684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.66:58065)
0929 19:05:34.701800 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:34.701830 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":76,"threads_started":1}
I20260929 19:05:34.826915 24109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:34.826724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.67:49381)
0929 19:05:34.826847 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:34.826877 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":69,"threads_started":1}
I20260929 19:05:35.682771 24110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:35.682565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.65:35771)
0929 19:05:35.682695 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:35.682728 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":71,"threads_started":1}
I20260929 19:05:35.702526 24111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:35.702282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.66:41891)
0929 19:05:35.702461 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:35.702490 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":129,"threads_started":1}
I20260929 19:05:35.827545 24112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:35.827351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.67:57251)
0929 19:05:35.827475 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:35.827506 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":69,"threads_started":1}
I20260929 19:05:36.683420 24113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:36.683195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.65:56605)
0929 19:05:36.683338 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:36.683374 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":58,"threads_started":1}
I20260929 19:05:36.703109 24114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:36.702912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.66:36803)
0929 19:05:36.703031 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:36.703069 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":75,"threads_started":1}
I20260929 19:05:36.828177 24115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:36.827974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.67:48805)
0929 19:05:36.828105 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:36.828139 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":68,"threads_started":1}
I20260929 19:05:37.110872 24116 raft_consensus.cc:482] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:05:37.110982 24116 raft_consensus.cc:504] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd62ed66bc1040bbbe53171a1d114f7d" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 44703 } } peers { permanent_uuid: "bbdc71811c3b4416a687f18009ebc913" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38085 } } peers { permanent_uuid: "ffeec491014e4d6b89512c09a168dc9b" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 46257 } }
I20260929 19:05:37.111183 24116 leader_election.cc:288] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fd62ed66bc1040bbbe53171a1d114f7d (127.8.158.67:44703), bbdc71811c3b4416a687f18009ebc913 (127.8.158.66:38085)
W20260929 19:05:37.111582 23701 leader_election.cc:341] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fd62ed66bc1040bbbe53171a1d114f7d (127.8.158.67:44703): Not found: Tablet not found: a07def7d19c64fefa26370cb3b004ee2
W20260929 19:05:37.111680 23699 leader_election.cc:341] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bbdc71811c3b4416a687f18009ebc913 (127.8.158.66:38085): Not found: Tablet not found: a07def7d19c64fefa26370cb3b004ee2
I20260929 19:05:37.111750 23699 leader_election.cc:302] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ffeec491014e4d6b89512c09a168dc9b; no voters: bbdc71811c3b4416a687f18009ebc913, fd62ed66bc1040bbbe53171a1d114f7d
I20260929 19:05:37.111840 24116 raft_consensus.cc:2726] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 19:05:37.684139 24117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:37.683932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.65:50065)
0929 19:05:37.684052 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:37.684087 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":62,"threads_started":1}
I20260929 19:05:37.703756 24118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:37.703581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.66:59463)
0929 19:05:37.703689 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:37.703720 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":56,"threads_started":1}
I20260929 19:05:37.828805 24119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:37.828608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.67:54273)
0929 19:05:37.828733 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:37.828766 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":55,"threads_started":1}
I20260929 19:05:38.684831 24120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:38.684625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.65:47373)
0929 19:05:38.684752 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:38.684787 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":63,"threads_started":1}
I20260929 19:05:38.704401 24121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:38.704193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.66:55797)
0929 19:05:38.704332 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:38.704360 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":81,"threads_started":1}
I20260929 19:05:38.829543 24122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:38.829311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.67:38347)
0929 19:05:38.829460 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:38.829503 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":59,"threads_started":1}
I20260929 19:05:39.685530 24123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:39.685331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.65:42467)
0929 19:05:39.685447 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:39.685484 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":56,"threads_started":1}
I20260929 19:05:39.704957 24124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:39.704773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.66:40767)
0929 19:05:39.704888 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:39.704921 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":53,"threads_started":1}
I20260929 19:05:39.830222 24125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:39.830007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.67:34691)
0929 19:05:39.830145 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:39.830182 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":56,"threads_started":1}
I20260929 19:05:39.984719 24126 raft_consensus.cc:482] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:05:39.984831 24126 raft_consensus.cc:504] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd62ed66bc1040bbbe53171a1d114f7d" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 44703 } } peers { permanent_uuid: "bbdc71811c3b4416a687f18009ebc913" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38085 } } peers { permanent_uuid: "ffeec491014e4d6b89512c09a168dc9b" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 46257 } }
I20260929 19:05:39.985013 24126 leader_election.cc:288] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fd62ed66bc1040bbbe53171a1d114f7d (127.8.158.67:44703), bbdc71811c3b4416a687f18009ebc913 (127.8.158.66:38085)
W20260929 19:05:39.985443 23701 leader_election.cc:341] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fd62ed66bc1040bbbe53171a1d114f7d (127.8.158.67:44703): Not found: Tablet not found: a07def7d19c64fefa26370cb3b004ee2
W20260929 19:05:39.985520 23699 leader_election.cc:341] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bbdc71811c3b4416a687f18009ebc913 (127.8.158.66:38085): Not found: Tablet not found: a07def7d19c64fefa26370cb3b004ee2
I20260929 19:05:39.985558 23699 leader_election.cc:302] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ffeec491014e4d6b89512c09a168dc9b; no voters: bbdc71811c3b4416a687f18009ebc913, fd62ed66bc1040bbbe53171a1d114f7d
I20260929 19:05:39.985620 24126 raft_consensus.cc:2726] T a07def7d19c64fefa26370cb3b004ee2 P ffeec491014e4d6b89512c09a168dc9b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 19:05:40.686228 24127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:40.686036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.65:50003)
0929 19:05:40.686151 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:40.686184 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":58,"threads_started":1}
I20260929 19:05:40.705583 24128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:40.705394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.66:36637)
0929 19:05:40.705519 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:40.705547 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":67,"threads_started":1}
I20260929 19:05:40.830835 24129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:40.830651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.67:36167)
0929 19:05:40.830761 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:40.830795 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":55,"threads_started":1}
I20260929 19:05:41.686986 24130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:41.686767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.65:40589)
0929 19:05:41.686903 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:41.686941 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":61,"threads_started":1}
I20260929 19:05:41.706147 24131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:41.705963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.66:49453)
0929 19:05:41.706073 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:41.706106 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":57,"threads_started":1}
I20260929 19:05:41.831480 24132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:41.831246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.67:47573)
0929 19:05:41.831392 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:41.831438 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":69,"threads_started":1}
I20260929 19:05:42.687911 24133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:42.687655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.65:45707)
0929 19:05:42.687814 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:42.687856 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":89,"threads_started":1}
I20260929 19:05:42.706800 24134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:42.706627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.66:36615)
0929 19:05:42.706731 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:42.706762 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":56,"threads_started":1}
I20260929 19:05:42.832242 24135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:42.832030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:38373 (local address 127.8.158.67:36575)
0929 19:05:42.832165 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:42.832200 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":70,"threads_started":1}
I20260929 19:05:42.838001 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 23654
I20260929 19:05:42.842463 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 23817
I20260929 19:05:42.846570 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 23950
2026-09-29T19:05:42Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18298 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:05:42 dist-test-slave-nmsz krb5kdc[24138](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:05:42 dist-test-slave-nmsz krb5kdc[24138](info): set up 2 sockets
Sep 29 19:05:42 dist-test-slave-nmsz krb5kdc[24138](info): commencing operation
krb5kdc: starting...
W20260929 19:05:43.914234 8825 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Sep 29 19:05:43 dist-test-slave-nmsz krb5kdc[24138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:05:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:05:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.158.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
I20260929 19:05:44.029992 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:35519
--webserver_interface=127.8.158.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:43293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:35519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:05:44.107278 24151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:05:44.107496 24151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:05:44.107532 24151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:05:44.108677 24151 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 19:05:44.108737 24151 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 19:05:44.108758 24151 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 19:05:44.108779 24151 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:05:44.108809 24151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:05:44.108834 24151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:05:44.108860 24151 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:05:44.108880 24151 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:05:44.110371 24151 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:43293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:35519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:35519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.158.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:05:44.110658 24151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:05:44.110874 24151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:05:44.115921 24156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:44.115980 24157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:05:44.116170 24151 server_base.cc:1034] running on GCE node
W20260929 19:05:44.116003 24159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:05:44.116511 24151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:05:44.116796 24151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:05:44.117961 24151 hybrid_clock.cc:648] HybridClock initialized: now 1790708744117946 us; error 42 us; skew 500 ppm
Sep 29 19:05:44 dist-test-slave-nmsz krb5kdc[24138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708744, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:05:44.120417 24151 init.cc:377] Logged in from keytab as kudu/127.8.158.126@KRBTEST.COM (short username kudu)
I20260929 19:05:44.120873 24151 webserver.cc:460] Webserver started at http://127.8.158.126:44353/ using document root <none> and password file <none>
I20260929 19:05:44.121102 24151 fs_manager.cc:359] Metadata directory not provided
I20260929 19:05:44.121166 24151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:05:44.121308 24151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:05:44.122411 24151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "443331b8c71b4bbea9b6cc9ff9bca0b0"
format_stamp: "Formatted at 2026-09-29 19:05:44 on dist-test-slave-nmsz"
server_key: "45f265def8124e8902fc2312f7e7e437"
server_key_iv: "8f3997a84fbf126c4b34f1fab07770fd"
server_key_version: "encryptionkey@0"
I20260929 19:05:44.122794 24151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "443331b8c71b4bbea9b6cc9ff9bca0b0"
format_stamp: "Formatted at 2026-09-29 19:05:44 on dist-test-slave-nmsz"
server_key: "45f265def8124e8902fc2312f7e7e437"
server_key_iv: "8f3997a84fbf126c4b34f1fab07770fd"
server_key_version: "encryptionkey@0"
I20260929 19:05:44.124125 24151 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:05:44.124806 24166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:44.125013 24151 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:05:44.125157 24151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "443331b8c71b4bbea9b6cc9ff9bca0b0"
format_stamp: "Formatted at 2026-09-29 19:05:44 on dist-test-slave-nmsz"
server_key: "45f265def8124e8902fc2312f7e7e437"
server_key_iv: "8f3997a84fbf126c4b34f1fab07770fd"
server_key_version: "encryptionkey@0"
I20260929 19:05:44.125242 24151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:05:44.143291 24151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:05:44.145076 24151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:05:44.145234 24151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:05:44.149143 24151 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.126:35519
I20260929 19:05:44.149179 24218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.126:35519 every 8 connection(s)
I20260929 19:05:44.149518 24151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:05:44.150372 24219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:05:44.152494 24219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0: Bootstrap starting.
I20260929 19:05:44.153059 24219 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0: Neither blocks nor log segments found. Creating new log.
I20260929 19:05:44.153337 24219 log.cc:824] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0: Log is configured to *not* fsync() on all Append() calls
I20260929 19:05:44.154131 24219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0: No bootstrap required, opened a new log
I20260929 19:05:44.154114 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 24151
I20260929 19:05:44.154214 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:05:44.154467 8825 external_mini_cluster.cc:1428] Setting key 6fd84ff4d23864a328d60938ddcdce1d
I20260929 19:05:44.156418 24219 raft_consensus.cc:348] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443331b8c71b4bbea9b6cc9ff9bca0b0" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35519 } }
I20260929 19:05:44.156598 24219 raft_consensus.cc:374] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:05:44.156635 24219 raft_consensus.cc:729] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443331b8c71b4bbea9b6cc9ff9bca0b0, State: Initialized, Role: FOLLOWER
I20260929 19:05:44.156776 24219 consensus_queue.cc:260] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443331b8c71b4bbea9b6cc9ff9bca0b0" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35519 } }
I20260929 19:05:44.156869 24219 raft_consensus.cc:388] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:05:44.156909 24219 raft_consensus.cc:482] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:05:44.156955 24219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:05:44.157610 24219 raft_consensus.cc:504] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443331b8c71b4bbea9b6cc9ff9bca0b0" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35519 } }
I20260929 19:05:44.157714 24219 leader_election.cc:302] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 443331b8c71b4bbea9b6cc9ff9bca0b0; no voters:
I20260929 19:05:44.157847 24219 leader_election.cc:288] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:05:44.157902 24224 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:05:44.158052 24219 sys_catalog.cc:564] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:05:44.158066 24224 raft_consensus.cc:686] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [term 1 LEADER]: Becoming Leader. State: Replica: 443331b8c71b4bbea9b6cc9ff9bca0b0, State: Running, Role: LEADER
I20260929 19:05:44.158187 24224 consensus_queue.cc:237] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443331b8c71b4bbea9b6cc9ff9bca0b0" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35519 } }
I20260929 19:05:44.158533 24225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "443331b8c71b4bbea9b6cc9ff9bca0b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443331b8c71b4bbea9b6cc9ff9bca0b0" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35519 } } }
I20260929 19:05:44.158533 24226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 443331b8c71b4bbea9b6cc9ff9bca0b0. Latest consensus state: current_term: 1 leader_uuid: "443331b8c71b4bbea9b6cc9ff9bca0b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443331b8c71b4bbea9b6cc9ff9bca0b0" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35519 } } }
I20260929 19:05:44.158625 24225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [sys.catalog]: This master's current role is: LEADER
I20260929 19:05:44.158634 24226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [sys.catalog]: This master's current role is: LEADER
I20260929 19:05:44.158900 24233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:05:44.159374 24233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 29 19:05:44 dist-test-slave-nmsz krb5kdc[24138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:05:44.163757 24233 catalog_manager.cc:1347] Generated new cluster ID: 7beb21957b384c0b96202505a0ea581c
I20260929 19:05:44.163811 24233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:05:44.166003 24223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:44.154980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51318 (local address 127.8.158.126:35519)
0929 19:05:44.155146 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:44.155155 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:44.155185 (+ 30us) server_negotiation.cc:407] Connection header received
0929 19:05:44.155656 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:44.155665 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:44.155723 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:44.155863 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:44.156458 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:44.158129 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:44.159390 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:44.159952 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:44.160087 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:44.162734 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:44.162738 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:44.162739 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:44.162749 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:44.164628 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:44.164958 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:44.164960 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:44.164960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:44.165063 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:44.165232 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:44.165232 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:44.165232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:44.165490 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:44.165603 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:44.165692 (+ 89us) server_negotiation.cc:299] Negotiation successful
0929 19:05:44.165810 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":76,"threads_started":1}
I20260929 19:05:44.176234 24233 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:05:44.176709 24233 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:05:44.184432 24233 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0: Generated new TSK 0
I20260929 19:05:44.184581 24233 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.158.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab.
Entry for principal kudu/127.8.158.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab.
I20260929 19:05:44.212935 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.65:0
--local_ip_for_outbound_sockets=127.8.158.65
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:35519
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab
--principal=kudu/127.8.158.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:43293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:05:44.292209 24245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:05:44.292395 24245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:05:44.292418 24245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:05:44.293520 24245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:05:44.293560 24245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:05:44.293604 24245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.65
I20260929 19:05:44.295060 24245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:43293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab
--principal=kudu/127.8.158.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:35519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.65
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:05:44.295293 24245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:05:44.295507 24245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:05:44.300215 24253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:44.300231 24251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:44.300264 24250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:05:44.300320 24245 server_base.cc:1034] running on GCE node
I20260929 19:05:44.300560 24245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:05:44.300778 24245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:05:44.301927 24245 hybrid_clock.cc:648] HybridClock initialized: now 1790708744301918 us; error 39 us; skew 500 ppm
Sep 29 19:05:44 dist-test-slave-nmsz krb5kdc[24138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708744, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:05:44.304297 24245 init.cc:377] Logged in from keytab as kudu/127.8.158.65@KRBTEST.COM (short username kudu)
I20260929 19:05:44.304718 24245 webserver.cc:460] Webserver started at http://127.8.158.65:36399/ using document root <none> and password file <none>
I20260929 19:05:44.304884 24245 fs_manager.cc:359] Metadata directory not provided
I20260929 19:05:44.304924 24245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:05:44.305007 24245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:05:44.305923 24245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-0/data/instance:
uuid: "ec0920ea49e24bbd938e0a19722d72f0"
format_stamp: "Formatted at 2026-09-29 19:05:44 on dist-test-slave-nmsz"
server_key: "bc1bdfe877c751866b10d0bd553cbbb5"
server_key_iv: "70443c5e45a036b800f7f0c61e068ddb"
server_key_version: "encryptionkey@0"
I20260929 19:05:44.306208 24245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance:
uuid: "ec0920ea49e24bbd938e0a19722d72f0"
format_stamp: "Formatted at 2026-09-29 19:05:44 on dist-test-slave-nmsz"
server_key: "bc1bdfe877c751866b10d0bd553cbbb5"
server_key_iv: "70443c5e45a036b800f7f0c61e068ddb"
server_key_version: "encryptionkey@0"
I20260929 19:05:44.307426 24245 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:05:44.308070 24260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:44.308337 24245 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:05:44.308420 24245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-0/wal
uuid: "ec0920ea49e24bbd938e0a19722d72f0"
format_stamp: "Formatted at 2026-09-29 19:05:44 on dist-test-slave-nmsz"
server_key: "bc1bdfe877c751866b10d0bd553cbbb5"
server_key_iv: "70443c5e45a036b800f7f0c61e068ddb"
server_key_version: "encryptionkey@0"
I20260929 19:05:44.308472 24245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:05:44.318135 24245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:05:44.319870 24245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:05:44.319973 24245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:05:44.320122 24245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:05:44.320406 24245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:05:44.320438 24245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:44.320475 24245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:05:44.320501 24245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:44.325882 24245 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.65:35205
I20260929 19:05:44.325919 24373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.65:35205 every 8 connection(s)
I20260929 19:05:44.326346 24245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
I20260929 19:05:44.327369 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 24245
I20260929 19:05:44.327471 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance
I20260929 19:05:44.327644 8825 external_mini_cluster.cc:1428] Setting key 9631f5c25ded7bac413afa977f16919f
Sep 29 19:05:44 dist-test-slave-nmsz krb5kdc[24138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708744, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.158.66@KRBTEST.COM; defaulting to no policy
I20260929 19:05:44.336483 24223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:44.327074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:36357 (local address 127.8.158.126:35519)
0929 19:05:44.327136 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:44.327139 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:44.327313 (+ 174us) server_negotiation.cc:407] Connection header received
0929 19:05:44.327907 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:44.327908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:44.327919 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:44.327988 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:44.328956 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:44.329736 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:44.331298 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:44.331485 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:44.331537 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:44.333779 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:44.333781 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:44.333781 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:44.333788 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:44.335168 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:44.335560 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:44.335561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:44.335562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:44.335621 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:44.335869 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:44.335870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:44.335871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:44.335982 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:44.336055 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:44.336310 (+ 255us) server_negotiation.cc:299] Negotiation successful
0929 19:05:44.336393 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 19:05:44.336663 24376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:44.327047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.65:36357)
0929 19:05:44.327207 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:44.327240 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:44.327813 (+ 573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:44.328038 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:44.328041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:44.328687 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:05:44.328873 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:44.328878 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:44.329825 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:44.329826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:44.331219 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:44.331222 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:44.331566 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:44.331567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:44.331712 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:44.332173 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:05:44.332183 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:05:44.333688 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:05:44.335267 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:44.335273 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:44.335275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:44.335486 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:44.335701 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:44.335702 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:44.335703 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:44.335805 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:44.336046 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:05:44.336052 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:05:44.336185 (+ 133us) client_negotiation.cc:769] Sending connection context
0929 19:05:44.336289 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 19:05:44.336432 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":84,"threads_started":1}
I20260929 19:05:44.337087 24374 heartbeater.cc:347] Connected to a master server at 127.8.158.126:35519
I20260929 19:05:44.337193 24374 heartbeater.cc:464] Registering TS with master...
I20260929 19:05:44.337368 24374 heartbeater.cc:511] Master 127.8.158.126:35519 requested a full tablet report, sending...
I20260929 19:05:44.337852 24183 ts_manager.cc:194] Registered new tserver with Master: ec0920ea49e24bbd938e0a19722d72f0 (127.8.158.65:35205)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.66@KRBTEST.COM" created.
I20260929 19:05:44.338539 24183 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.65@KRBTEST.COM'} at 127.8.158.65:36357
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab.
Entry for principal kudu/127.8.158.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab.
I20260929 19:05:44.352859 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.66:0
--local_ip_for_outbound_sockets=127.8.158.66
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:35519
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab
--principal=kudu/127.8.158.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:43293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:05:44.431373 24379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:05:44.431597 24379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:05:44.431644 24379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:05:44.432771 24379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:05:44.432833 24379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:05:44.432880 24379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.66
I20260929 19:05:44.434338 24379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:43293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab
--principal=kudu/127.8.158.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:35519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.66
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:05:44.434589 24379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:05:44.434808 24379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:05:44.439546 24387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:44.439581 24385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:05:44.439730 24379 server_base.cc:1034] running on GCE node
W20260929 19:05:44.439548 24384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:05:44.439950 24379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:05:44.440174 24379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:05:44.441329 24379 hybrid_clock.cc:648] HybridClock initialized: now 1790708744441326 us; error 38 us; skew 500 ppm
Sep 29 19:05:44 dist-test-slave-nmsz krb5kdc[24138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708744, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:05:44.443758 24379 init.cc:377] Logged in from keytab as kudu/127.8.158.66@KRBTEST.COM (short username kudu)
I20260929 19:05:44.444180 24379 webserver.cc:460] Webserver started at http://127.8.158.66:43479/ using document root <none> and password file <none>
I20260929 19:05:44.444397 24379 fs_manager.cc:359] Metadata directory not provided
I20260929 19:05:44.444442 24379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:05:44.444559 24379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:05:44.445596 24379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-1/data/instance:
uuid: "2692934c705841b5ac802c10868baf47"
format_stamp: "Formatted at 2026-09-29 19:05:44 on dist-test-slave-nmsz"
server_key: "37e781166689b43d535f360e36283177"
server_key_iv: "acbf5dab4566e76de08ad740536bf57c"
server_key_version: "encryptionkey@0"
I20260929 19:05:44.445938 24379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance:
uuid: "2692934c705841b5ac802c10868baf47"
format_stamp: "Formatted at 2026-09-29 19:05:44 on dist-test-slave-nmsz"
server_key: "37e781166689b43d535f360e36283177"
server_key_iv: "acbf5dab4566e76de08ad740536bf57c"
server_key_version: "encryptionkey@0"
I20260929 19:05:44.447271 24379 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:05:44.448033 24394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:44.448212 24379 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:05:44.448310 24379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-1/wal
uuid: "2692934c705841b5ac802c10868baf47"
format_stamp: "Formatted at 2026-09-29 19:05:44 on dist-test-slave-nmsz"
server_key: "37e781166689b43d535f360e36283177"
server_key_iv: "acbf5dab4566e76de08ad740536bf57c"
server_key_version: "encryptionkey@0"
I20260929 19:05:44.448380 24379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:05:44.464427 24379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:05:44.466651 24379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:05:44.466796 24379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:05:44.467000 24379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:05:44.467271 24379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:05:44.467322 24379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:44.467361 24379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:05:44.467386 24379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:44.472929 24379 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.66:32803
I20260929 19:05:44.472977 24507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.66:32803 every 8 connection(s)
I20260929 19:05:44.473254 24379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
I20260929 19:05:44.476986 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 24379
I20260929 19:05:44.477087 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance
I20260929 19:05:44.477278 8825 external_mini_cluster.cc:1428] Setting key 1dcdab3c4ca39e1779751c241c021b5d
Sep 29 19:05:44 dist-test-slave-nmsz krb5kdc[24138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708744, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.66@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:05:44.482553 24223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:44.473877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:43883 (local address 127.8.158.126:35519)
0929 19:05:44.473947 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:44.473950 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:44.474097 (+ 147us) server_negotiation.cc:407] Connection header received
0929 19:05:44.474637 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:44.474638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:44.474648 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:44.474700 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:44.475557 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:44.476052 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:44.477235 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:44.477465 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:44.477532 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:44.479849 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:44.479851 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:44.479851 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:44.479858 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:44.481211 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:44.481618 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:44.481619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:44.481620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:44.481676 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:44.481923 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:44.481924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:44.481924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:44.482032 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:44.482105 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:44.482384 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 19:05:44.482461 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 19:05:44.482769 24510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:44.473839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.66:43883)
0929 19:05:44.473989 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:44.474022 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:44.474548 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:44.474747 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:44.474750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:44.475286 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:05:44.475454 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:44.475458 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:44.476168 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:44.476169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:44.477137 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:44.477140 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:44.477553 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:44.477554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:44.477661 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:44.478187 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:05:44.478195 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:05:44.479758 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:05:44.481307 (+ 1549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:44.481314 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:44.481316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:44.481532 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:44.481748 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:44.481749 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:44.481750 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:44.481857 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:44.482096 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:05:44.482102 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:05:44.482239 (+ 137us) client_negotiation.cc:769] Sending connection context
0929 19:05:44.482363 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 19:05:44.482516 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
I20260929 19:05:44.483249 24508 heartbeater.cc:347] Connected to a master server at 127.8.158.126:35519
I20260929 19:05:44.483346 24508 heartbeater.cc:464] Registering TS with master...
I20260929 19:05:44.483505 24508 heartbeater.cc:511] Master 127.8.158.126:35519 requested a full tablet report, sending...
I20260929 19:05:44.483850 24183 ts_manager.cc:194] Registered new tserver with Master: 2692934c705841b5ac802c10868baf47 (127.8.158.66:32803)
I20260929 19:05:44.484372 24183 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.66@KRBTEST.COM'} at 127.8.158.66:43883
WARNING: no policy specified for kudu/127.8.158.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab.
Entry for principal kudu/127.8.158.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab.
I20260929 19:05:44.501051 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.67:0
--local_ip_for_outbound_sockets=127.8.158.67
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:35519
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab
--principal=kudu/127.8.158.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:43293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:05:44.578665 24513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:05:44.578886 24513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:05:44.578923 24513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:05:44.580027 24513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:05:44.580088 24513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:05:44.580133 24513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.67
I20260929 19:05:44.581543 24513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:43293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab
--principal=kudu/127.8.158.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:35519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.67
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:05:44.581801 24513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:05:44.582007 24513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:05:44.586515 24521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:44.586688 24518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:44.586719 24519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:05:44.587114 24513 server_base.cc:1034] running on GCE node
I20260929 19:05:44.587309 24513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:05:44.587530 24513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:05:44.588677 24513 hybrid_clock.cc:648] HybridClock initialized: now 1790708744588680 us; error 42 us; skew 500 ppm
Sep 29 19:05:44 dist-test-slave-nmsz krb5kdc[24138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708744, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:05:44.590989 24513 init.cc:377] Logged in from keytab as kudu/127.8.158.67@KRBTEST.COM (short username kudu)
I20260929 19:05:44.591428 24513 webserver.cc:460] Webserver started at http://127.8.158.67:36083/ using document root <none> and password file <none>
I20260929 19:05:44.591640 24513 fs_manager.cc:359] Metadata directory not provided
I20260929 19:05:44.591701 24513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:05:44.591859 24513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:05:44.592799 24513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-2/data/instance:
uuid: "1182f2efaaae4dfbab1b8ad73a06610b"
format_stamp: "Formatted at 2026-09-29 19:05:44 on dist-test-slave-nmsz"
server_key: "3da672f1d83b864f3adea6c98b9cf023"
server_key_iv: "2857cbb5cec791b823d0043eec7440d1"
server_key_version: "encryptionkey@0"
I20260929 19:05:44.593128 24513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance:
uuid: "1182f2efaaae4dfbab1b8ad73a06610b"
format_stamp: "Formatted at 2026-09-29 19:05:44 on dist-test-slave-nmsz"
server_key: "3da672f1d83b864f3adea6c98b9cf023"
server_key_iv: "2857cbb5cec791b823d0043eec7440d1"
server_key_version: "encryptionkey@0"
I20260929 19:05:44.594398 24513 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:05:44.595134 24528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:44.595302 24513 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:05:44.595391 24513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-2/wal
uuid: "1182f2efaaae4dfbab1b8ad73a06610b"
format_stamp: "Formatted at 2026-09-29 19:05:44 on dist-test-slave-nmsz"
server_key: "3da672f1d83b864f3adea6c98b9cf023"
server_key_iv: "2857cbb5cec791b823d0043eec7440d1"
server_key_version: "encryptionkey@0"
I20260929 19:05:44.595468 24513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:05:44.610512 24513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:05:44.612321 24513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:05:44.612469 24513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:05:44.612665 24513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:05:44.612939 24513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:05:44.612988 24513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:44.613026 24513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:05:44.613056 24513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:44.618521 24513 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.67:45145
I20260929 19:05:44.618573 24641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.67:45145 every 8 connection(s)
I20260929 19:05:44.618848 24513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
Sep 29 19:05:44 dist-test-slave-nmsz krb5kdc[24138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708744, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.67@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:05:44.625046 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 24513
I20260929 19:05:44.625128 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance
I20260929 19:05:44.625304 8825 external_mini_cluster.cc:1428] Setting key 178c58dbf211ac6510f48ce3a1b6da09
I20260929 19:05:44.627408 24223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:44.619517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:56385 (local address 127.8.158.126:35519)
0929 19:05:44.619563 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:44.619565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:44.619718 (+ 153us) server_negotiation.cc:407] Connection header received
0929 19:05:44.620156 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:44.620158 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:44.620166 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:44.620221 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:44.621037 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:44.621519 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:44.622452 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:44.622631 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:44.622690 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:44.624634 (+ 1944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:44.624636 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:44.624636 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:44.624643 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:44.626158 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:44.626677 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:44.626678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:44.626679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:44.626728 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:44.626900 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:44.626901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:44.626901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:44.626994 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:44.627042 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:44.627283 (+ 241us) server_negotiation.cc:299] Negotiation successful
0929 19:05:44.627348 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 19:05:44.627605 24644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:44.619473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.67:56385)
0929 19:05:44.619613 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:44.619651 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:44.620080 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:44.620268 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:44.620271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:44.620795 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:05:44.620964 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:44.620968 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:44.621593 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:44.621594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:44.622383 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:44.622386 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:44.622697 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:44.622698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:44.622799 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:44.623179 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:05:44.623186 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:05:44.624531 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:05:44.626238 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:44.626243 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:44.626244 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:44.626611 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:44.626768 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:44.626769 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:44.626769 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:44.626854 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:44.627041 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:05:44.627044 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:05:44.627183 (+ 139us) client_negotiation.cc:769] Sending connection context
0929 19:05:44.627261 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 19:05:44.627408 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":80,"threads_started":1}
I20260929 19:05:44.628028 24642 heartbeater.cc:347] Connected to a master server at 127.8.158.126:35519
I20260929 19:05:44.628121 24642 heartbeater.cc:464] Registering TS with master...
I20260929 19:05:44.628280 24642 heartbeater.cc:511] Master 127.8.158.126:35519 requested a full tablet report, sending...
I20260929 19:05:44.628625 24183 ts_manager.cc:194] Registered new tserver with Master: 1182f2efaaae4dfbab1b8ad73a06610b (127.8.158.67:45145)
I20260929 19:05:44.628978 24183 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.67@KRBTEST.COM'} at 127.8.158.67:56385
I20260929 19:05:44.636785 8825 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:05:44.643584 24223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:44.638051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51332 (local address 127.8.158.126:35519)
0929 19:05:44.638090 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:44.638091 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:44.638175 (+ 84us) server_negotiation.cc:407] Connection header received
0929 19:05:44.638213 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:44.638214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:44.638221 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:44.638268 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:44.638811 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:44.639261 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:44.640066 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:44.640245 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:44.640307 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:44.641325 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:44.641326 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:44.641326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:44.641332 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:44.642754 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:44.643029 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:44.643030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:44.643030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:44.643077 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:44.643216 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:44.643216 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:44.643216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:44.643325 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:44.643397 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:44.643476 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 19:05:44.643530 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 19:05:44.644884 24183 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51332:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:05:44.646528 24183 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:05:44.653529 24653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:44.648938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:35205 (local address 127.0.0.1:38432)
0929 19:05:44.649251 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:44.649263 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:44.649324 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:44.649610 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:44.649611 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:44.649613 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:05:44.649653 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:44.649655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:44.651171 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:44.651172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:44.652294 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:44.652297 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:44.653200 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:44.653201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:44.653255 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:44.653272 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 19:05:44.653347 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 19:05:44.653411 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":69,"threads_started":1}
I20260929 19:05:44.653554 24652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:44.648808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41330 (local address 127.8.158.66:32803)
0929 19:05:44.648999 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:44.649004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:44.649163 (+ 159us) server_negotiation.cc:407] Connection header received
0929 19:05:44.649341 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:44.649344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:44.649363 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:44.649459 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:05:44.649725 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:44.650563 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:44.652159 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:44.652972 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:44.653055 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:44.653118 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:44.653410 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 19:05:44.653459 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":88,"threads_started":1}
I20260929 19:05:44.653888 24654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:44.649054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38432 (local address 127.8.158.65:35205)
0929 19:05:44.649248 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:44.649253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:44.649387 (+ 134us) server_negotiation.cc:407] Connection header received
0929 19:05:44.649430 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:44.649433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:44.649449 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:44.649521 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:05:44.649724 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:44.651048 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:44.652387 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:44.652998 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:44.653095 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:44.653163 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:44.653750 (+ 587us) server_negotiation.cc:299] Negotiation successful
0929 19:05:44.653797 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":97,"threads_started":1}
I20260929 19:05:44.653821 24655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:44.649075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48490 (local address 127.8.158.67:45145)
0929 19:05:44.649395 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:44.649400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:44.649410 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:05:44.649447 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:44.649450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:44.649466 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:44.649542 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:05:44.649761 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:44.651007 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:44.652453 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:44.653213 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:44.653284 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:44.653335 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:44.653572 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 19:05:44.653742 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":76,"threads_started":1}
I20260929 19:05:44.653538 24650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:44.648708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.66:32803 (local address 127.0.0.1:41330)
0929 19:05:44.649002 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:44.649043 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:44.649187 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:44.649553 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:44.649556 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:44.649558 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:05:44.649623 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:44.649624 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:44.650667 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:44.650668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:44.652028 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:44.652031 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:44.653076 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:44.653077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:44.653252 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:44.653272 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 19:05:44.653381 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 19:05:44.653416 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":9,"thread_start_us":74,"threads_started":1}
I20260929 19:05:44.654552 24651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:44.648740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.67:45145 (local address 127.0.0.1:48490)
0929 19:05:44.649225 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:44.649235 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:44.649281 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:44.649620 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:44.649621 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:44.649623 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:05:44.649675 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:44.649677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:44.651114 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:44.651115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:44.652375 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:44.652378 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:44.653299 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:44.653300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:44.653373 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:44.653378 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 19:05:44.653436 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 19:05:44.653477 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":61,"threads_started":1}
I20260929 19:05:44.654752 24442 tablet_service.cc:1467] Processing CreateTablet for tablet 739d257ad894400bb3954184b554c623 (DEFAULT_TABLE table=test-table [id=590a323f98d44ba4829fd509dbe4a701]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:05:44.654768 24308 tablet_service.cc:1467] Processing CreateTablet for tablet 739d257ad894400bb3954184b554c623 (DEFAULT_TABLE table=test-table [id=590a323f98d44ba4829fd509dbe4a701]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:05:44.655133 24442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 739d257ad894400bb3954184b554c623. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:05:44.655133 24308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 739d257ad894400bb3954184b554c623. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:05:44.655220 24576 tablet_service.cc:1467] Processing CreateTablet for tablet 739d257ad894400bb3954184b554c623 (DEFAULT_TABLE table=test-table [id=590a323f98d44ba4829fd509dbe4a701]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:05:44.655517 24576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 739d257ad894400bb3954184b554c623. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:05:44.658021 24656 tablet_bootstrap.cc:492] T 739d257ad894400bb3954184b554c623 P 2692934c705841b5ac802c10868baf47: Bootstrap starting.
I20260929 19:05:44.658735 24657 tablet_bootstrap.cc:492] T 739d257ad894400bb3954184b554c623 P ec0920ea49e24bbd938e0a19722d72f0: Bootstrap starting.
I20260929 19:05:44.658972 24658 tablet_bootstrap.cc:492] T 739d257ad894400bb3954184b554c623 P 1182f2efaaae4dfbab1b8ad73a06610b: Bootstrap starting.
I20260929 19:05:44.659265 24656 tablet_bootstrap.cc:654] T 739d257ad894400bb3954184b554c623 P 2692934c705841b5ac802c10868baf47: Neither blocks nor log segments found. Creating new log.
I20260929 19:05:44.659500 24657 tablet_bootstrap.cc:654] T 739d257ad894400bb3954184b554c623 P ec0920ea49e24bbd938e0a19722d72f0: Neither blocks nor log segments found. Creating new log.
I20260929 19:05:44.659708 24658 tablet_bootstrap.cc:654] T 739d257ad894400bb3954184b554c623 P 1182f2efaaae4dfbab1b8ad73a06610b: Neither blocks nor log segments found. Creating new log.
I20260929 19:05:44.659968 24656 log.cc:824] T 739d257ad894400bb3954184b554c623 P 2692934c705841b5ac802c10868baf47: Log is configured to *not* fsync() on all Append() calls
I20260929 19:05:44.660144 24657 log.cc:824] T 739d257ad894400bb3954184b554c623 P ec0920ea49e24bbd938e0a19722d72f0: Log is configured to *not* fsync() on all Append() calls
I20260929 19:05:44.660313 24658 log.cc:824] T 739d257ad894400bb3954184b554c623 P 1182f2efaaae4dfbab1b8ad73a06610b: Log is configured to *not* fsync() on all Append() calls
I20260929 19:05:44.660782 24656 tablet_bootstrap.cc:492] T 739d257ad894400bb3954184b554c623 P 2692934c705841b5ac802c10868baf47: No bootstrap required, opened a new log
I20260929 19:05:44.660895 24656 ts_tablet_manager.cc:1397] T 739d257ad894400bb3954184b554c623 P 2692934c705841b5ac802c10868baf47: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260929 19:05:44.661000 24657 tablet_bootstrap.cc:492] T 739d257ad894400bb3954184b554c623 P ec0920ea49e24bbd938e0a19722d72f0: No bootstrap required, opened a new log
I20260929 19:05:44.661074 24657 ts_tablet_manager.cc:1397] T 739d257ad894400bb3954184b554c623 P ec0920ea49e24bbd938e0a19722d72f0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:05:44.661098 24658 tablet_bootstrap.cc:492] T 739d257ad894400bb3954184b554c623 P 1182f2efaaae4dfbab1b8ad73a06610b: No bootstrap required, opened a new log
I20260929 19:05:44.661160 24658 ts_tablet_manager.cc:1397] T 739d257ad894400bb3954184b554c623 P 1182f2efaaae4dfbab1b8ad73a06610b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:05:44.662774 24658 raft_consensus.cc:348] T 739d257ad894400bb3954184b554c623 P 1182f2efaaae4dfbab1b8ad73a06610b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2692934c705841b5ac802c10868baf47" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 32803 } } peers { permanent_uuid: "1182f2efaaae4dfbab1b8ad73a06610b" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45145 } } peers { permanent_uuid: "ec0920ea49e24bbd938e0a19722d72f0" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 35205 } }
I20260929 19:05:44.662935 24658 raft_consensus.cc:374] T 739d257ad894400bb3954184b554c623 P 1182f2efaaae4dfbab1b8ad73a06610b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:05:44.663000 24658 raft_consensus.cc:729] T 739d257ad894400bb3954184b554c623 P 1182f2efaaae4dfbab1b8ad73a06610b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1182f2efaaae4dfbab1b8ad73a06610b, State: Initialized, Role: FOLLOWER
I20260929 19:05:44.663130 24658 consensus_queue.cc:260] T 739d257ad894400bb3954184b554c623 P 1182f2efaaae4dfbab1b8ad73a06610b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2692934c705841b5ac802c10868baf47" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 32803 } } peers { permanent_uuid: "1182f2efaaae4dfbab1b8ad73a06610b" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45145 } } peers { permanent_uuid: "ec0920ea49e24bbd938e0a19722d72f0" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 35205 } }
I20260929 19:05:44.663213 24657 raft_consensus.cc:348] T 739d257ad894400bb3954184b554c623 P ec0920ea49e24bbd938e0a19722d72f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2692934c705841b5ac802c10868baf47" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 32803 } } peers { permanent_uuid: "1182f2efaaae4dfbab1b8ad73a06610b" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45145 } } peers { permanent_uuid: "ec0920ea49e24bbd938e0a19722d72f0" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 35205 } }
I20260929 19:05:44.663199 24656 raft_consensus.cc:348] T 739d257ad894400bb3954184b554c623 P 2692934c705841b5ac802c10868baf47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2692934c705841b5ac802c10868baf47" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 32803 } } peers { permanent_uuid: "1182f2efaaae4dfbab1b8ad73a06610b" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45145 } } peers { permanent_uuid: "ec0920ea49e24bbd938e0a19722d72f0" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 35205 } }
I20260929 19:05:44.663386 24656 raft_consensus.cc:374] T 739d257ad894400bb3954184b554c623 P 2692934c705841b5ac802c10868baf47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:05:44.663386 24657 raft_consensus.cc:374] T 739d257ad894400bb3954184b554c623 P ec0920ea49e24bbd938e0a19722d72f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:05:44.663435 24658 ts_tablet_manager.cc:1428] T 739d257ad894400bb3954184b554c623 P 1182f2efaaae4dfbab1b8ad73a06610b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 19:05:44.663434 24657 raft_consensus.cc:729] T 739d257ad894400bb3954184b554c623 P ec0920ea49e24bbd938e0a19722d72f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec0920ea49e24bbd938e0a19722d72f0, State: Initialized, Role: FOLLOWER
I20260929 19:05:44.663475 24656 raft_consensus.cc:729] T 739d257ad894400bb3954184b554c623 P 2692934c705841b5ac802c10868baf47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2692934c705841b5ac802c10868baf47, State: Initialized, Role: FOLLOWER
I20260929 19:05:44.663568 24642 heartbeater.cc:503] Master 127.8.158.126:35519 was elected leader, sending a full tablet report...
I20260929 19:05:44.663566 24657 consensus_queue.cc:260] T 739d257ad894400bb3954184b554c623 P ec0920ea49e24bbd938e0a19722d72f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2692934c705841b5ac802c10868baf47" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 32803 } } peers { permanent_uuid: "1182f2efaaae4dfbab1b8ad73a06610b" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45145 } } peers { permanent_uuid: "ec0920ea49e24bbd938e0a19722d72f0" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 35205 } }
I20260929 19:05:44.663614 24656 consensus_queue.cc:260] T 739d257ad894400bb3954184b554c623 P 2692934c705841b5ac802c10868baf47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2692934c705841b5ac802c10868baf47" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 32803 } } peers { permanent_uuid: "1182f2efaaae4dfbab1b8ad73a06610b" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45145 } } peers { permanent_uuid: "ec0920ea49e24bbd938e0a19722d72f0" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 35205 } }
I20260929 19:05:44.663864 24657 ts_tablet_manager.cc:1428] T 739d257ad894400bb3954184b554c623 P ec0920ea49e24bbd938e0a19722d72f0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 19:05:44.663908 24656 ts_tablet_manager.cc:1428] T 739d257ad894400bb3954184b554c623 P 2692934c705841b5ac802c10868baf47: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 19:05:44.663892 24374 heartbeater.cc:503] Master 127.8.158.126:35519 was elected leader, sending a full tablet report...
I20260929 19:05:44.664011 24508 heartbeater.cc:503] Master 127.8.158.126:35519 was elected leader, sending a full tablet report...
W20260929 19:05:44.723780 24509 tablet.cc:2369] T 739d257ad894400bb3954184b554c623 P 2692934c705841b5ac802c10868baf47: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 19:05:44.827301 24375 tablet.cc:2369] T 739d257ad894400bb3954184b554c623 P ec0920ea49e24bbd938e0a19722d72f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 19:05:44.869342 24643 tablet.cc:2369] T 739d257ad894400bb3954184b554c623 P 1182f2efaaae4dfbab1b8ad73a06610b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:05:45.066285 24662 raft_consensus.cc:482] T 739d257ad894400bb3954184b554c623 P 1182f2efaaae4dfbab1b8ad73a06610b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:05:45.066423 24662 raft_consensus.cc:504] T 739d257ad894400bb3954184b554c623 P 1182f2efaaae4dfbab1b8ad73a06610b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2692934c705841b5ac802c10868baf47" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 32803 } } peers { permanent_uuid: "1182f2efaaae4dfbab1b8ad73a06610b" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45145 } } peers { permanent_uuid: "ec0920ea49e24bbd938e0a19722d72f0" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 35205 } }
I20260929 19:05:45.066756 24662 leader_election.cc:288] T 739d257ad894400bb3954184b554c623 P 1182f2efaaae4dfbab1b8ad73a06610b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2692934c705841b5ac802c10868baf47 (127.8.158.66:32803), ec0920ea49e24bbd938e0a19722d72f0 (127.8.158.65:35205)
I20260929 19:05:45.070868 24654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:45.067060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:50763 (local address 127.8.158.65:35205)
0929 19:05:45.067127 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:45.067130 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:45.067141 (+ 11us) server_negotiation.cc:407] Connection header received
0929 19:05:45.067178 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:45.067179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:45.067202 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:45.067272 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:05:45.067461 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:45.068230 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:45.069797 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:45.070461 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:45.070550 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:45.070559 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:45.070749 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 19:05:45.070791 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 19:05:45.070868 24652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:45.067074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:44567 (local address 127.8.158.66:32803)
0929 19:05:45.067135 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:45.067137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:45.067237 (+ 100us) server_negotiation.cc:407] Connection header received
0929 19:05:45.067265 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:45.067266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:45.067275 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:45.067330 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:05:45.067479 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:45.068230 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:45.069797 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:45.070461 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:45.070549 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:45.070559 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:45.070749 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 19:05:45.070790 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 19:05:45.070904 24644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:45.066951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:35205 (local address 127.8.158.67:50763)
0929 19:05:45.067010 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:45.067023 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:45.067085 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:45.067283 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:45.067283 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:45.067285 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:05:45.067373 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:45.067375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:45.068329 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:45.068331 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:45.069674 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:45.069677 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:45.070568 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:45.070569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:45.070668 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:45.070670 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 19:05:45.070731 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 19:05:45.070806 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":112}
I20260929 19:05:45.070910 24665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:45.066960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.66:32803 (local address 127.8.158.67:44567)
0929 19:05:45.067142 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:45.067152 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:45.067201 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:45.067343 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:45.067344 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:45.067345 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:05:45.067411 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:45.067412 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:45.068329 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:45.068331 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:45.069674 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:45.069677 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:45.070568 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:45.070569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:45.070663 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:45.070668 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 19:05:45.070731 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 19:05:45.070800 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":101,"thread_start_us":94,"threads_started":1}
I20260929 19:05:45.071259 24328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "739d257ad894400bb3954184b554c623" candidate_uuid: "1182f2efaaae4dfbab1b8ad73a06610b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec0920ea49e24bbd938e0a19722d72f0" is_pre_election: true
I20260929 19:05:45.071435 24328 raft_consensus.cc:2454] T 739d257ad894400bb3954184b554c623 P ec0920ea49e24bbd938e0a19722d72f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1182f2efaaae4dfbab1b8ad73a06610b in term 0.
I20260929 19:05:45.071520 24462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "739d257ad894400bb3954184b554c623" candidate_uuid: "1182f2efaaae4dfbab1b8ad73a06610b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2692934c705841b5ac802c10868baf47" is_pre_election: true
I20260929 19:05:45.071646 24462 raft_consensus.cc:2454] T 739d257ad894400bb3954184b554c623 P 2692934c705841b5ac802c10868baf47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1182f2efaaae4dfbab1b8ad73a06610b in term 0.
I20260929 19:05:45.071663 24529 leader_election.cc:302] T 739d257ad894400bb3954184b554c623 P 1182f2efaaae4dfbab1b8ad73a06610b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1182f2efaaae4dfbab1b8ad73a06610b, ec0920ea49e24bbd938e0a19722d72f0; no voters:
I20260929 19:05:45.071792 24662 raft_consensus.cc:2781] T 739d257ad894400bb3954184b554c623 P 1182f2efaaae4dfbab1b8ad73a06610b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:05:45.071847 24662 raft_consensus.cc:482] T 739d257ad894400bb3954184b554c623 P 1182f2efaaae4dfbab1b8ad73a06610b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:05:45.071882 24662 raft_consensus.cc:3037] T 739d257ad894400bb3954184b554c623 P 1182f2efaaae4dfbab1b8ad73a06610b [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:05:45.072602 24662 raft_consensus.cc:504] T 739d257ad894400bb3954184b554c623 P 1182f2efaaae4dfbab1b8ad73a06610b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2692934c705841b5ac802c10868baf47" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 32803 } } peers { permanent_uuid: "1182f2efaaae4dfbab1b8ad73a06610b" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45145 } } peers { permanent_uuid: "ec0920ea49e24bbd938e0a19722d72f0" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 35205 } }
I20260929 19:05:45.072748 24662 leader_election.cc:288] T 739d257ad894400bb3954184b554c623 P 1182f2efaaae4dfbab1b8ad73a06610b [CANDIDATE]: Term 1 election: Requested vote from peers 2692934c705841b5ac802c10868baf47 (127.8.158.66:32803), ec0920ea49e24bbd938e0a19722d72f0 (127.8.158.65:35205)
I20260929 19:05:45.072929 24328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "739d257ad894400bb3954184b554c623" candidate_uuid: "1182f2efaaae4dfbab1b8ad73a06610b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec0920ea49e24bbd938e0a19722d72f0"
I20260929 19:05:45.072952 24462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "739d257ad894400bb3954184b554c623" candidate_uuid: "1182f2efaaae4dfbab1b8ad73a06610b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2692934c705841b5ac802c10868baf47"
I20260929 19:05:45.073005 24328 raft_consensus.cc:3037] T 739d257ad894400bb3954184b554c623 P ec0920ea49e24bbd938e0a19722d72f0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:05:45.073069 24462 raft_consensus.cc:3037] T 739d257ad894400bb3954184b554c623 P 2692934c705841b5ac802c10868baf47 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:05:45.073787 24328 raft_consensus.cc:2454] T 739d257ad894400bb3954184b554c623 P ec0920ea49e24bbd938e0a19722d72f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1182f2efaaae4dfbab1b8ad73a06610b in term 1.
I20260929 19:05:45.073906 24462 raft_consensus.cc:2454] T 739d257ad894400bb3954184b554c623 P 2692934c705841b5ac802c10868baf47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1182f2efaaae4dfbab1b8ad73a06610b in term 1.
I20260929 19:05:45.073945 24529 leader_election.cc:302] T 739d257ad894400bb3954184b554c623 P 1182f2efaaae4dfbab1b8ad73a06610b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1182f2efaaae4dfbab1b8ad73a06610b, ec0920ea49e24bbd938e0a19722d72f0; no voters:
I20260929 19:05:45.074018 24662 raft_consensus.cc:2781] T 739d257ad894400bb3954184b554c623 P 1182f2efaaae4dfbab1b8ad73a06610b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:05:45.074163 24662 raft_consensus.cc:686] T 739d257ad894400bb3954184b554c623 P 1182f2efaaae4dfbab1b8ad73a06610b [term 1 LEADER]: Becoming Leader. State: Replica: 1182f2efaaae4dfbab1b8ad73a06610b, State: Running, Role: LEADER
I20260929 19:05:45.074262 24662 consensus_queue.cc:237] T 739d257ad894400bb3954184b554c623 P 1182f2efaaae4dfbab1b8ad73a06610b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2692934c705841b5ac802c10868baf47" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 32803 } } peers { permanent_uuid: "1182f2efaaae4dfbab1b8ad73a06610b" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45145 } } peers { permanent_uuid: "ec0920ea49e24bbd938e0a19722d72f0" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 35205 } }
I20260929 19:05:45.075088 24182 catalog_manager.cc:5495] T 739d257ad894400bb3954184b554c623 P 1182f2efaaae4dfbab1b8ad73a06610b reported cstate change: term changed from 0 to 1, leader changed from <none> to 1182f2efaaae4dfbab1b8ad73a06610b (127.8.158.67). New cstate: current_term: 1 leader_uuid: "1182f2efaaae4dfbab1b8ad73a06610b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2692934c705841b5ac802c10868baf47" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 32803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1182f2efaaae4dfbab1b8ad73a06610b" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec0920ea49e24bbd938e0a19722d72f0" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 35205 } health_report { overall_health: UNKNOWN } } }
I20260929 19:05:45.093362 24655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:45.091055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48502 (local address 127.8.158.67:45145)
0929 19:05:45.091112 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:45.091114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:45.091124 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:05:45.091155 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:45.091156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:45.091175 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:45.091257 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:05:45.091376 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:45.091841 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:45.092588 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:45.092797 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:45.092870 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:45.093023 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:05:45.093129 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:05:45.093218 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:45.093238 (+ 20us) server_negotiation.cc:299] Negotiation successful
0929 19:05:45.093258 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 19:05:45.095073 24462 raft_consensus.cc:1261] T 739d257ad894400bb3954184b554c623 P 2692934c705841b5ac802c10868baf47 [term 1 FOLLOWER]: Refusing update from remote peer 1182f2efaaae4dfbab1b8ad73a06610b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:05:45.095287 24328 raft_consensus.cc:1261] T 739d257ad894400bb3954184b554c623 P ec0920ea49e24bbd938e0a19722d72f0 [term 1 FOLLOWER]: Refusing update from remote peer 1182f2efaaae4dfbab1b8ad73a06610b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:05:45.095383 24666 consensus_queue.cc:1035] T 739d257ad894400bb3954184b554c623 P 1182f2efaaae4dfbab1b8ad73a06610b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2692934c705841b5ac802c10868baf47" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 32803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:05:45.095563 24662 consensus_queue.cc:1035] T 739d257ad894400bb3954184b554c623 P 1182f2efaaae4dfbab1b8ad73a06610b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec0920ea49e24bbd938e0a19722d72f0" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 35205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:05:45.098141 24671 mvcc.cc:205] Tried to move back new op lower bound from 7334743019906916352 to 7334743019824910336. Current Snapshot: MvccSnapshot[applied={T|T < 7334743019906916352 or (T in {7334743019906916352})}]
I20260929 19:05:45.099838 24182 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:05:45.099929 24182 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:05:45.100632 24182 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0: Sending DeleteTablet for 3 replicas of tablet 739d257ad894400bb3954184b554c623
I20260929 19:05:45.100917 24576 tablet_service.cc:1514] Processing DeleteTablet for tablet 739d257ad894400bb3954184b554c623 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:05:45 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:48490
I20260929 19:05:45.100956 24442 tablet_service.cc:1514] Processing DeleteTablet for tablet 739d257ad894400bb3954184b554c623 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:05:45 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:41330
I20260929 19:05:45.100987 24308 tablet_service.cc:1514] Processing DeleteTablet for tablet 739d257ad894400bb3954184b554c623 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:05:45 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:38432
I20260929 19:05:45.101022 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 24245
I20260929 19:05:45.101651 24678 tablet_replica.cc:330] T 739d257ad894400bb3954184b554c623 P 1182f2efaaae4dfbab1b8ad73a06610b: stopping tablet replica
I20260929 19:05:45.101780 24678 raft_consensus.cc:2229] T 739d257ad894400bb3954184b554c623 P 1182f2efaaae4dfbab1b8ad73a06610b [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:05:45.101902 24678 raft_consensus.cc:2258] T 739d257ad894400bb3954184b554c623 P 1182f2efaaae4dfbab1b8ad73a06610b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:05:45.102352 24678 ts_tablet_manager.cc:1905] T 739d257ad894400bb3954184b554c623 P 1182f2efaaae4dfbab1b8ad73a06610b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:05:45.103286 24679 tablet_replica.cc:330] T 739d257ad894400bb3954184b554c623 P 2692934c705841b5ac802c10868baf47: stopping tablet replica
I20260929 19:05:45.103421 24679 raft_consensus.cc:2229] T 739d257ad894400bb3954184b554c623 P 2692934c705841b5ac802c10868baf47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:05:45.103574 24679 raft_consensus.cc:2258] T 739d257ad894400bb3954184b554c623 P 2692934c705841b5ac802c10868baf47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:05:45.103972 24678 ts_tablet_manager.cc:1918] T 739d257ad894400bb3954184b554c623 P 1182f2efaaae4dfbab1b8ad73a06610b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:05:45.104058 24679 ts_tablet_manager.cc:1905] T 739d257ad894400bb3954184b554c623 P 2692934c705841b5ac802c10868baf47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:05:45.104099 24678 log.cc:1196] T 739d257ad894400bb3954184b554c623 P 1182f2efaaae4dfbab1b8ad73a06610b: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-2/wal/wals/739d257ad894400bb3954184b554c623
I20260929 19:05:45.104355 24678 ts_tablet_manager.cc:1939] T 739d257ad894400bb3954184b554c623 P 1182f2efaaae4dfbab1b8ad73a06610b: Deleting consensus metadata
I20260929 19:05:45.104755 24167 catalog_manager.cc:4841] TS 1182f2efaaae4dfbab1b8ad73a06610b (127.8.158.67:45145): tablet 739d257ad894400bb3954184b554c623 (table test-table [id=590a323f98d44ba4829fd509dbe4a701]) successfully deleted
I20260929 19:05:45.106077 24679 ts_tablet_manager.cc:1918] T 739d257ad894400bb3954184b554c623 P 2692934c705841b5ac802c10868baf47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:05:45.106153 24679 log.cc:1196] T 739d257ad894400bb3954184b554c623 P 2692934c705841b5ac802c10868baf47: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-1/wal/wals/739d257ad894400bb3954184b554c623
I20260929 19:05:45.106410 24679 ts_tablet_manager.cc:1939] T 739d257ad894400bb3954184b554c623 P 2692934c705841b5ac802c10868baf47: Deleting consensus metadata
I20260929 19:05:45.106838 24168 catalog_manager.cc:4841] TS 2692934c705841b5ac802c10868baf47 (127.8.158.66:32803): tablet 739d257ad894400bb3954184b554c623 (table test-table [id=590a323f98d44ba4829fd509dbe4a701]) successfully deleted
W20260929 19:05:45.107416 24167 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.158.65:35205 (error 108)
I20260929 19:05:45.107889 24651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:45.107763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:35205 (local address 127.0.0.1:38434)
0929 19:05:45.107820 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:45.107858 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.65:35205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 19:05:45.107918 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 24379
W20260929 19:05:45.108858 24167 catalog_manager.cc:4584] TS ec0920ea49e24bbd938e0a19722d72f0 (127.8.158.65:35205): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 739d257ad894400bb3954184b554c623: Network error: Client connection negotiation failed: client connection to 127.8.158.65:35205: connect: Connection refused (error 111)
I20260929 19:05:45.112627 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 24513
I20260929 19:05:45.117214 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 24151
I20260929 19:06:05.122275 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:35519
--webserver_interface=127.8.158.126
--webserver_port=44353
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:43293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:35519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:06:05.201753 24681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:06:05.201937 24681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:06:05.201958 24681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:06:05.203110 24681 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 19:06:05.203147 24681 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 19:06:05.203161 24681 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 19:06:05.203172 24681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:06:05.203186 24681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:06:05.203198 24681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:06:05.203253 24681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:06:05.203270 24681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:06:05.204672 24681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:43293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:35519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:35519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.158.126
--webserver_port=44353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:06:05.204906 24681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:06:05.205147 24681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 19:06:05.213683 24681 server_base.cc:1034] running on GCE node
W20260929 19:06:05.213750 24688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:05.213645 24690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:05.213629 24687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:05.214126 24681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:06:05.214491 24681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:06:05.215669 24681 hybrid_clock.cc:648] HybridClock initialized: now 1790708765215665 us; error 46 us; skew 500 ppm
Sep 29 19:06:05 dist-test-slave-nmsz krb5kdc[24138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708765, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:06:05.218300 24681 init.cc:377] Logged in from keytab as kudu/127.8.158.126@KRBTEST.COM (short username kudu)
I20260929 19:06:05.218820 24681 webserver.cc:460] Webserver started at http://127.8.158.126:44353/ using document root <none> and password file <none>
I20260929 19:06:05.219061 24681 fs_manager.cc:359] Metadata directory not provided
I20260929 19:06:05.219151 24681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:06:05.220510 24681 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:06:05.221163 24697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:05.221331 24681 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:05.221441 24681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "443331b8c71b4bbea9b6cc9ff9bca0b0"
format_stamp: "Formatted at 2026-09-29 19:05:44 on dist-test-slave-nmsz"
server_key: "45f265def8124e8902fc2312f7e7e437"
server_key_iv: "8f3997a84fbf126c4b34f1fab07770fd"
server_key_version: "encryptionkey@0"
I20260929 19:06:05.221832 24681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:06:05.231144 24681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:06:05.232877 24681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:06:05.233028 24681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:06:05.236985 24681 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.126:35519
I20260929 19:06:05.237036 24749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.126:35519 every 8 connection(s)
I20260929 19:06:05.237463 24681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:06:05.238902 24750 sys_catalog.cc:263] Verifying existing consensus state
I20260929 19:06:05.239478 24750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0: Bootstrap starting.
I20260929 19:06:05.247210 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 24681
I20260929 19:06:05.247503 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.65:35205
--local_ip_for_outbound_sockets=127.8.158.65
--tserver_master_addrs=127.8.158.126:35519
--webserver_port=36399
--webserver_interface=127.8.158.65
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab
--principal=kudu/127.8.158.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:43293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:06:05.247699 24750 log.cc:824] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0: Log is configured to *not* fsync() on all Append() calls
I20260929 19:06:05.249778 24750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 19:06:05.250082 24750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0: Bootstrap complete.
I20260929 19:06:05.253980 24750 raft_consensus.cc:348] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443331b8c71b4bbea9b6cc9ff9bca0b0" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35519 } }
I20260929 19:06:05.254396 24750 raft_consensus.cc:729] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443331b8c71b4bbea9b6cc9ff9bca0b0, State: Initialized, Role: FOLLOWER
I20260929 19:06:05.254570 24750 consensus_queue.cc:260] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443331b8c71b4bbea9b6cc9ff9bca0b0" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35519 } }
I20260929 19:06:05.254659 24750 raft_consensus.cc:388] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:06:05.254712 24750 raft_consensus.cc:482] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:06:05.254760 24750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [term 1 FOLLOWER]: Advancing to term 2
I20260929 19:06:05.256331 24750 raft_consensus.cc:504] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443331b8c71b4bbea9b6cc9ff9bca0b0" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35519 } }
I20260929 19:06:05.256453 24750 leader_election.cc:302] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 443331b8c71b4bbea9b6cc9ff9bca0b0; no voters:
I20260929 19:06:05.256619 24750 leader_election.cc:288] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 19:06:05.256691 24754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 19:06:05.256886 24754 raft_consensus.cc:686] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [term 2 LEADER]: Becoming Leader. State: Replica: 443331b8c71b4bbea9b6cc9ff9bca0b0, State: Running, Role: LEADER
I20260929 19:06:05.256894 24750 sys_catalog.cc:564] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:06:05.257027 24754 consensus_queue.cc:237] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443331b8c71b4bbea9b6cc9ff9bca0b0" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35519 } }
I20260929 19:06:05.257364 24755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "443331b8c71b4bbea9b6cc9ff9bca0b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443331b8c71b4bbea9b6cc9ff9bca0b0" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35519 } } }
I20260929 19:06:05.257362 24756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 443331b8c71b4bbea9b6cc9ff9bca0b0. Latest consensus state: current_term: 2 leader_uuid: "443331b8c71b4bbea9b6cc9ff9bca0b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443331b8c71b4bbea9b6cc9ff9bca0b0" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35519 } } }
I20260929 19:06:05.257468 24755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [sys.catalog]: This master's current role is: LEADER
I20260929 19:06:05.257472 24756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [sys.catalog]: This master's current role is: LEADER
I20260929 19:06:05.257931 24760 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:06:05.258787 24760 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:06:05.258888 24760 catalog_manager.cc:1259] Loaded cluster ID: 7beb21957b384c0b96202505a0ea581c
I20260929 19:06:05.258931 24760 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:06:05.259809 24760 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:06:05.260000 24760 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0: Loaded TSK: 0
I20260929 19:06:05.276296 24760 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0: Generated new TSK 1
I20260929 19:06:05.276386 24760 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 19:06:05.329493 24753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:06:05.329685 24753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:06:05.329708 24753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:06:05.330950 24753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:06:05.330991 24753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:06:05.331017 24753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.65
I20260929 19:06:05.332495 24753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:43293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab
--principal=kudu/127.8.158.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.65:35205
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.158.65
--webserver_port=36399
--tserver_master_addrs=127.8.158.126:35519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.65
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:06:05.332739 24753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:06:05.332979 24753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:06:05.338742 24780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:05.338783 24777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:05.338771 24778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:05.339139 24753 server_base.cc:1034] running on GCE node
I20260929 19:06:05.339339 24753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:06:05.339570 24753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:06:05.340744 24753 hybrid_clock.cc:648] HybridClock initialized: now 1790708765340729 us; error 31 us; skew 500 ppm
Sep 29 19:06:05 dist-test-slave-nmsz krb5kdc[24138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708765, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:06:05.343210 24753 init.cc:377] Logged in from keytab as kudu/127.8.158.65@KRBTEST.COM (short username kudu)
I20260929 19:06:05.343669 24753 webserver.cc:460] Webserver started at http://127.8.158.65:36399/ using document root <none> and password file <none>
I20260929 19:06:05.343916 24753 fs_manager.cc:359] Metadata directory not provided
I20260929 19:06:05.343971 24753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:06:05.345240 24753 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:06:05.345923 24787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:05.346122 24753 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:06:05.346216 24753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-0/wal
uuid: "ec0920ea49e24bbd938e0a19722d72f0"
format_stamp: "Formatted at 2026-09-29 19:05:44 on dist-test-slave-nmsz"
server_key: "bc1bdfe877c751866b10d0bd553cbbb5"
server_key_iv: "70443c5e45a036b800f7f0c61e068ddb"
server_key_version: "encryptionkey@0"
I20260929 19:06:05.346581 24753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:06:05.358898 24753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:06:05.361065 24753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:06:05.361222 24753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:06:05.361449 24753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:06:05.361881 24794 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 19:06:05.363358 24753 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 19:06:05.363413 24753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260929 19:06:05.363437 24753 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 19:06:05.364046 24753 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 19:06:05.364091 24753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260929 19:06:05.364130 24794 tablet_bootstrap.cc:492] T 739d257ad894400bb3954184b554c623 P ec0920ea49e24bbd938e0a19722d72f0: Bootstrap starting.
I20260929 19:06:05.371747 24753 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.65:35205
I20260929 19:06:05.371845 24901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.65:35205 every 8 connection(s)
I20260929 19:06:05.372345 24753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
I20260929 19:06:05.372846 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 24753
I20260929 19:06:05.373148 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.66:32803
--local_ip_for_outbound_sockets=127.8.158.66
--tserver_master_addrs=127.8.158.126:35519
--webserver_port=43479
--webserver_interface=127.8.158.66
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab
--principal=kudu/127.8.158.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:43293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:06:05.375800 24794 log.cc:824] T 739d257ad894400bb3954184b554c623 P ec0920ea49e24bbd938e0a19722d72f0: Log is configured to *not* fsync() on all Append() calls
I20260929 19:06:05.379555 24794 tablet_bootstrap.cc:492] T 739d257ad894400bb3954184b554c623 P ec0920ea49e24bbd938e0a19722d72f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 19:06:05.379928 24794 tablet_bootstrap.cc:492] T 739d257ad894400bb3954184b554c623 P ec0920ea49e24bbd938e0a19722d72f0: Bootstrap complete.
I20260929 19:06:05.381306 24794 ts_tablet_manager.cc:1397] T 739d257ad894400bb3954184b554c623 P ec0920ea49e24bbd938e0a19722d72f0: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.004s
I20260929 19:06:05.382656 24794 raft_consensus.cc:348] T 739d257ad894400bb3954184b554c623 P ec0920ea49e24bbd938e0a19722d72f0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2692934c705841b5ac802c10868baf47" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 32803 } } peers { permanent_uuid: "1182f2efaaae4dfbab1b8ad73a06610b" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45145 } } peers { permanent_uuid: "ec0920ea49e24bbd938e0a19722d72f0" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 35205 } }
I20260929 19:06:05.383087 24794 raft_consensus.cc:729] T 739d257ad894400bb3954184b554c623 P ec0920ea49e24bbd938e0a19722d72f0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec0920ea49e24bbd938e0a19722d72f0, State: Initialized, Role: FOLLOWER
I20260929 19:06:05.383291 24794 consensus_queue.cc:260] T 739d257ad894400bb3954184b554c623 P ec0920ea49e24bbd938e0a19722d72f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2692934c705841b5ac802c10868baf47" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 32803 } } peers { permanent_uuid: "1182f2efaaae4dfbab1b8ad73a06610b" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45145 } } peers { permanent_uuid: "ec0920ea49e24bbd938e0a19722d72f0" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 35205 } }
I20260929 19:06:05.383585 24794 ts_tablet_manager.cc:1428] T 739d257ad894400bb3954184b554c623 P ec0920ea49e24bbd938e0a19722d72f0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
Sep 29 19:06:05 dist-test-slave-nmsz krb5kdc[24138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708765, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:06:05.388943 24905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:05.373025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.65:44997)
0929 19:06:05.375840 (+ 2815us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:05.375887 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:05.376468 (+ 581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:05.377111 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:05.377116 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:05.377524 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:06:05.377806 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:05.377813 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:05.379685 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:05.379686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:05.381112 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:05.381115 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:05.382142 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:05.382143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:05.382335 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:05.382997 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:06:05.383007 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:06:05.384906 (+ 1899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:06:05.387135 (+ 2229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:06:05.387141 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:06:05.387143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:06:05.387459 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:06:05.387747 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:06:05.387748 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:06:05.387748 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:06:05.387862 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:06:05.388313 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:06:05.388318 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:06:05.388461 (+ 143us) client_negotiation.cc:769] Sending connection context
0929 19:06:05.388569 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 19:06:05.388743 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2771,"thread_start_us":1377,"threads_started":1}
I20260929 19:06:05.388991 24904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:05.374193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:44997 (local address 127.8.158.126:35519)
0929 19:06:05.376673 (+ 2480us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:05.376682 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:05.376713 (+ 31us) server_negotiation.cc:407] Connection header received
0929 19:06:05.376837 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:05.376844 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:05.376896 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:05.377053 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:05.377964 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:05.379523 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:05.381243 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:05.381907 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:05.381992 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:05.385032 (+ 3040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:05.385036 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:05.385037 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:06:05.385048 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:05.387035 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:05.387560 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:05.387564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:05.387564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:05.387673 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:05.387931 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:05.387932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:05.387933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:05.388228 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:05.388316 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:05.388641 (+ 325us) server_negotiation.cc:299] Negotiation successful
0929 19:06:05.388772 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2431,"thread_start_us":92,"threads_started":1}
I20260929 19:06:05.390249 24902 heartbeater.cc:347] Connected to a master server at 127.8.158.126:35519
I20260929 19:06:05.390362 24902 heartbeater.cc:464] Registering TS with master...
I20260929 19:06:05.390527 24902 heartbeater.cc:511] Master 127.8.158.126:35519 requested a full tablet report, sending...
I20260929 19:06:05.391237 24714 ts_manager.cc:194] Registered new tserver with Master: ec0920ea49e24bbd938e0a19722d72f0 (127.8.158.65:35205)
I20260929 19:06:05.392405 24714 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.65@KRBTEST.COM'} at 127.8.158.65:44997
I20260929 19:06:05.393395 24902 heartbeater.cc:503] Master 127.8.158.126:35519 was elected leader, sending a full tablet report...
I20260929 19:06:05.398492 24909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:05.391845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:35205 (local address 127.0.0.1:47550)
0929 19:06:05.394423 (+ 2578us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:05.394457 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:05.394521 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:05.394792 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:05.394795 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:05.394798 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:06:05.394857 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:05.394859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:05.396056 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:05.396057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:05.397323 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:05.397326 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:05.398183 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:05.398184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:05.398271 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:05.398284 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 19:06:05.398375 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 19:06:05.398422 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2550,"thread_start_us":80,"threads_started":1}
I20260929 19:06:05.398494 24910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:05.392082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47550 (local address 127.8.158.65:35205)
0929 19:06:05.394639 (+ 2557us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:05.394643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:05.394651 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:06:05.394680 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:05.394682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:05.394695 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:05.394764 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:06:05.394936 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:05.395960 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:05.397403 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:05.398092 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:05.398166 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:05.398211 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:05.398388 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 19:06:05.398426 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2530,"thread_start_us":70,"threads_started":1}
I20260929 19:06:05.399214 24836 tablet_service.cc:1514] Processing DeleteTablet for tablet 739d257ad894400bb3954184b554c623 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:05:45 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:47550
I20260929 19:06:05.401654 24911 tablet_replica.cc:330] stopping tablet replica
I20260929 19:06:05.401736 24911 raft_consensus.cc:2229] T 739d257ad894400bb3954184b554c623 P ec0920ea49e24bbd938e0a19722d72f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:06:05.401827 24911 raft_consensus.cc:2258] T 739d257ad894400bb3954184b554c623 P ec0920ea49e24bbd938e0a19722d72f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:06:05.402108 24911 ts_tablet_manager.cc:1905] T 739d257ad894400bb3954184b554c623 P ec0920ea49e24bbd938e0a19722d72f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:06:05.404289 24911 ts_tablet_manager.cc:1918] T 739d257ad894400bb3954184b554c623 P ec0920ea49e24bbd938e0a19722d72f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:06:05.404351 24911 log.cc:1196] T 739d257ad894400bb3954184b554c623 P ec0920ea49e24bbd938e0a19722d72f0: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-0/wal/wals/739d257ad894400bb3954184b554c623
I20260929 19:06:05.404597 24911 ts_tablet_manager.cc:1939] T 739d257ad894400bb3954184b554c623 P ec0920ea49e24bbd938e0a19722d72f0: Deleting consensus metadata
I20260929 19:06:05.405077 24698 catalog_manager.cc:4841] TS ec0920ea49e24bbd938e0a19722d72f0 (127.8.158.65:35205): tablet 739d257ad894400bb3954184b554c623 (table test-table [id=590a323f98d44ba4829fd509dbe4a701]) successfully deleted
W20260929 19:06:05.459344 24906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:06:05.459553 24906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:06:05.459589 24906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:06:05.460729 24906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:06:05.460794 24906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:06:05.460839 24906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.66
I20260929 19:06:05.462261 24906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:43293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab
--principal=kudu/127.8.158.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.66:32803
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.158.66
--webserver_port=43479
--tserver_master_addrs=127.8.158.126:35519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.66
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:06:05.462523 24906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:06:05.462740 24906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:06:05.467414 24916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:05.467473 24917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:05.467450 24919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:05.467715 24906 server_base.cc:1034] running on GCE node
I20260929 19:06:05.467891 24906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:06:05.468063 24906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:06:05.469241 24906 hybrid_clock.cc:648] HybridClock initialized: now 1790708765469229 us; error 27 us; skew 500 ppm
Sep 29 19:06:05 dist-test-slave-nmsz krb5kdc[24138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708765, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:06:05.471658 24906 init.cc:377] Logged in from keytab as kudu/127.8.158.66@KRBTEST.COM (short username kudu)
I20260929 19:06:05.472079 24906 webserver.cc:460] Webserver started at http://127.8.158.66:43479/ using document root <none> and password file <none>
I20260929 19:06:05.472265 24906 fs_manager.cc:359] Metadata directory not provided
I20260929 19:06:05.472314 24906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:06:05.473498 24906 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:06:05.474040 24926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:05.474268 24906 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 19:06:05.474388 24906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-1/wal
uuid: "2692934c705841b5ac802c10868baf47"
format_stamp: "Formatted at 2026-09-29 19:05:44 on dist-test-slave-nmsz"
server_key: "37e781166689b43d535f360e36283177"
server_key_iv: "acbf5dab4566e76de08ad740536bf57c"
server_key_version: "encryptionkey@0"
I20260929 19:06:05.474725 24906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:06:05.490814 24906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:06:05.492965 24906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:06:05.493095 24906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:06:05.493398 24906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:06:05.493722 24906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:06:05.493784 24906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:05.493830 24906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:06:05.493861 24906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:05.500077 24906 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.66:32803
I20260929 19:06:05.500114 25039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.66:32803 every 8 connection(s)
I20260929 19:06:05.500476 24906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
Sep 29 19:06:05 dist-test-slave-nmsz krb5kdc[24138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708765, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.66@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:06:05.508731 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 24906
I20260929 19:06:05.509023 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.67:45145
--local_ip_for_outbound_sockets=127.8.158.67
--tserver_master_addrs=127.8.158.126:35519
--webserver_port=36083
--webserver_interface=127.8.158.67
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab
--principal=kudu/127.8.158.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:43293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:06:05.511348 24904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:05.501362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:54787 (local address 127.8.158.126:35519)
0929 19:06:05.501567 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:05.501570 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:05.501764 (+ 194us) server_negotiation.cc:407] Connection header received
0929 19:06:05.502418 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:05.502419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:05.502434 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:05.502493 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:05.503549 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:05.504091 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:05.505224 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:05.505448 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:05.505539 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:05.507937 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:05.507939 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:05.507939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:06:05.507946 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:05.509597 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:05.510116 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:05.510117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:05.510118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:05.510210 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:05.510530 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:05.510532 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:05.510532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:05.510715 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:05.510810 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:05.511108 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 19:06:05.511217 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
I20260929 19:06:05.511361 25042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:05.501269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.66:54787)
0929 19:06:05.501590 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:05.501674 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:05.502254 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:05.502563 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:05.502568 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:05.503205 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:06:05.503404 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:05.503409 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:05.504198 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:05.504199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:05.505102 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:05.505105 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:05.505552 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:05.505553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:05.505689 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:05.506267 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:06:05.506276 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:06:05.507796 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:06:05.509683 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:06:05.509690 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:06:05.509692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:06:05.510028 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:06:05.510302 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:06:05.510303 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:06:05.510303 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:06:05.510457 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:06:05.510827 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:06:05.510833 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:06:05.510952 (+ 119us) client_negotiation.cc:769] Sending connection context
0929 19:06:05.511042 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 19:06:05.511167 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":99,"threads_started":1}
I20260929 19:06:05.512002 25040 heartbeater.cc:347] Connected to a master server at 127.8.158.126:35519
I20260929 19:06:05.512115 25040 heartbeater.cc:464] Registering TS with master...
I20260929 19:06:05.512318 25040 heartbeater.cc:511] Master 127.8.158.126:35519 requested a full tablet report, sending...
I20260929 19:06:05.512687 24714 ts_manager.cc:194] Registered new tserver with Master: 2692934c705841b5ac802c10868baf47 (127.8.158.66:32803)
I20260929 19:06:05.513072 24714 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.66@KRBTEST.COM'} at 127.8.158.66:54787
W20260929 19:06:05.593185 25043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:06:05.593403 25043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:06:05.593441 25043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:06:05.594614 25043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:06:05.594671 25043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:06:05.594717 25043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.67
I20260929 19:06:05.596127 25043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:43293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab
--principal=kudu/127.8.158.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.67:45145
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.158.67
--webserver_port=36083
--tserver_master_addrs=127.8.158.126:35519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.67
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:06:05.596385 25043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:06:05.596603 25043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:06:05.601320 25048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:05.601359 25051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:05.601320 25049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:05.601516 25043 server_base.cc:1034] running on GCE node
I20260929 19:06:05.601758 25043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:06:05.601994 25043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:06:05.603152 25043 hybrid_clock.cc:648] HybridClock initialized: now 1790708765603138 us; error 27 us; skew 500 ppm
Sep 29 19:06:05 dist-test-slave-nmsz krb5kdc[24138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708765, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:06:05.605535 25043 init.cc:377] Logged in from keytab as kudu/127.8.158.67@KRBTEST.COM (short username kudu)
I20260929 19:06:05.605958 25043 webserver.cc:460] Webserver started at http://127.8.158.67:36083/ using document root <none> and password file <none>
I20260929 19:06:05.606172 25043 fs_manager.cc:359] Metadata directory not provided
I20260929 19:06:05.606228 25043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:06:05.607586 25043 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:06:05.608191 25058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:05.608430 25043 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:06:05.608522 25043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-2/wal
uuid: "1182f2efaaae4dfbab1b8ad73a06610b"
format_stamp: "Formatted at 2026-09-29 19:05:44 on dist-test-slave-nmsz"
server_key: "3da672f1d83b864f3adea6c98b9cf023"
server_key_iv: "2857cbb5cec791b823d0043eec7440d1"
server_key_version: "encryptionkey@0"
I20260929 19:06:05.608938 25043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:06:05.623087 25043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:06:05.625305 25043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:06:05.625458 25043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:06:05.625674 25043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:06:05.625946 25043 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:06:05.625998 25043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:05.626036 25043 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:06:05.626065 25043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:05.631718 25043 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.67:45145
I20260929 19:06:05.631759 25171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.67:45145 every 8 connection(s)
I20260929 19:06:05.632059 25043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
I20260929 19:06:05.634934 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 25043
Sep 29 19:06:05 dist-test-slave-nmsz krb5kdc[24138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708765, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.67@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:06:05.642195 24904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:05.632679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:38453 (local address 127.8.158.126:35519)
0929 19:06:05.632759 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:05.632761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:05.632951 (+ 190us) server_negotiation.cc:407] Connection header received
0929 19:06:05.633377 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:05.633378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:05.633387 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:05.633433 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:05.634267 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:05.634776 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:05.635946 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:05.636149 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:05.636237 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:05.638304 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:05.638305 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:05.638306 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:06:05.638339 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:05.640358 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:05.641079 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:05.641081 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:05.641081 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:05.641145 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:05.641468 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:05.641469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:05.641469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:05.641599 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:05.641677 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:05.642010 (+ 333us) server_negotiation.cc:299] Negotiation successful
0929 19:06:05.642103 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 19:06:05.642529 25174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:05.632664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.67:38453)
0929 19:06:05.632839 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:05.632872 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:05.633300 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:05.633487 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:05.633490 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:05.634022 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:06:05.634194 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:05.634198 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:05.634879 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:05.634880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:05.635842 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:05.635845 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:05.636247 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:05.636248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:05.636372 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:05.636840 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:06:05.636848 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:06:05.638211 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:06:05.640595 (+ 2384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:06:05.640603 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:06:05.640605 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:06:05.640964 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:06:05.641228 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:06:05.641229 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:06:05.641230 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:06:05.641379 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:06:05.641676 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:06:05.641680 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:06:05.641852 (+ 172us) client_negotiation.cc:769] Sending connection context
0929 19:06:05.641974 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 19:06:05.642206 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":93,"threads_started":1}
I20260929 19:06:05.643098 25172 heartbeater.cc:347] Connected to a master server at 127.8.158.126:35519
I20260929 19:06:05.643172 25176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:05.635308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40332 (local address 127.8.158.126:35519)
0929 19:06:05.635416 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:05.635418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:05.635498 (+ 80us) server_negotiation.cc:407] Connection header received
0929 19:06:05.635575 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:05.635575 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:05.635586 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:05.635643 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:05.636220 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:05.636774 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:05.637812 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:05.638082 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:05.638176 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:05.639482 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:05.639483 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:05.639484 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:06:05.639491 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:05.641339 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:05.641787 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:05.641788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:05.641789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:05.641855 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:05.642103 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:05.642104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:05.642105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:05.642262 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:05.642474 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:05.642967 (+ 493us) server_negotiation.cc:299] Negotiation successful
0929 19:06:05.643060 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":57,"threads_started":1}
I20260929 19:06:05.643195 25172 heartbeater.cc:464] Registering TS with master...
I20260929 19:06:05.643369 25172 heartbeater.cc:511] Master 127.8.158.126:35519 requested a full tablet report, sending...
I20260929 19:06:05.643711 24714 ts_manager.cc:194] Registered new tserver with Master: 1182f2efaaae4dfbab1b8ad73a06610b (127.8.158.67:45145)
I20260929 19:06:05.644160 8825 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:06:05.644263 24714 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.67@KRBTEST.COM'} at 127.8.158.67:38453
I20260929 19:06:05.647090 25176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:05.644557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40346 (local address 127.8.158.126:35519)
0929 19:06:05.644601 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:05.644602 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:05.644711 (+ 109us) server_negotiation.cc:407] Connection header received
0929 19:06:05.644787 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:05.644788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:05.644820 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:05.644912 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:06:05.645059 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:05.645494 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:05.646233 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:05.646521 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:05.646611 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:05.646817 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:06:05.646863 (+ 46us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 19:06:05.647011 (+ 148us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40346: token expired
Metrics: {"server-negotiator.queue_time_us":23}
W20260929 19:06:05.647303 25176 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40346: token expired
W20260929 19:06:05.647390 24646 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.8.158.126:35519)
I20260929 19:06:05.669112 25176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:05.652105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40354 (local address 127.8.158.126:35519)
0929 19:06:05.656734 (+ 4629us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:05.656736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:05.656746 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:06:05.656781 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:05.656782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:05.656791 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:05.656854 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:05.659729 (+ 2875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:05.660625 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:05.662419 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:05.662718 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:05.662778 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:05.665943 (+ 3165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:05.665945 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:05.665945 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:06:05.665951 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:05.667831 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:05.668183 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:05.668183 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:05.668184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:05.668229 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:05.668383 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:05.668384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:05.668384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:05.668534 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:05.668613 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:05.668937 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 19:06:05.669009 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4602}
I20260929 19:06:05.670734 24714 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40354:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:06:05.673851 24714 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:06:05.676424 24836 tablet_service.cc:1467] Processing CreateTablet for tablet 2f6c4df30be3407f8bf8225c25c09a78 (DEFAULT_TABLE table=test-table [id=1dc0d5a03ad04941b7f38f74ff6c75bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:06:05.676620 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f6c4df30be3407f8bf8225c25c09a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:06:05.678483 24794 tablet_bootstrap.cc:492] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0: Bootstrap starting.
I20260929 19:06:05.678931 24794 tablet_bootstrap.cc:654] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0: Neither blocks nor log segments found. Creating new log.
I20260929 19:06:05.679523 24794 tablet_bootstrap.cc:492] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0: No bootstrap required, opened a new log
I20260929 19:06:05.679594 24794 ts_tablet_manager.cc:1397] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 19:06:05.679747 24794 raft_consensus.cc:348] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2692934c705841b5ac802c10868baf47" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 32803 } } peers { permanent_uuid: "1182f2efaaae4dfbab1b8ad73a06610b" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45145 } } peers { permanent_uuid: "ec0920ea49e24bbd938e0a19722d72f0" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 35205 } }
I20260929 19:06:05.679838 24794 raft_consensus.cc:374] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:06:05.679884 24794 raft_consensus.cc:729] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec0920ea49e24bbd938e0a19722d72f0, State: Initialized, Role: FOLLOWER
I20260929 19:06:05.679940 24794 consensus_queue.cc:260] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2692934c705841b5ac802c10868baf47" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 32803 } } peers { permanent_uuid: "1182f2efaaae4dfbab1b8ad73a06610b" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45145 } } peers { permanent_uuid: "ec0920ea49e24bbd938e0a19722d72f0" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 35205 } }
I20260929 19:06:05.680037 24794 ts_tablet_manager.cc:1428] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:05.681211 25178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:05.675990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.67:45145 (local address 127.0.0.1:35486)
0929 19:06:05.676140 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:05.676175 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:05.676219 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:05.676736 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:05.676737 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:05.676740 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:06:05.676788 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:05.676790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:05.678507 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:05.678508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:05.679730 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:05.679733 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:05.680856 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:05.680857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:05.681029 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:05.681032 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:06:05.681102 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 19:06:05.681137 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":96,"threads_started":1}
I20260929 19:06:05.681229 25180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:05.676289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35486 (local address 127.8.158.67:45145)
0929 19:06:05.676480 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:05.676486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:05.676500 (+ 14us) server_negotiation.cc:407] Connection header received
0929 19:06:05.676550 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:05.676553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:05.676572 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:05.676675 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:06:05.676873 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:05.678360 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:05.679804 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:05.680729 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:05.680922 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:05.681007 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:05.681132 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 19:06:05.681168 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":99,"threads_started":1}
I20260929 19:06:05.681911 25106 tablet_service.cc:1467] Processing CreateTablet for tablet 2f6c4df30be3407f8bf8225c25c09a78 (DEFAULT_TABLE table=test-table [id=1dc0d5a03ad04941b7f38f74ff6c75bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:06:05.682255 25106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f6c4df30be3407f8bf8225c25c09a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:06:05.682343 24909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:05.675897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.66:32803 (local address 127.0.0.1:58830)
0929 19:06:05.675941 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:05.675951 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:05.675987 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:05.678900 (+ 2913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:05.678901 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:05.678902 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:06:05.678931 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:05.678933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:05.680235 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:05.680236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:05.681223 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:05.681225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:05.682126 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:05.682127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:05.682183 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:05.682187 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:06:05.682218 (+ 31us) client_negotiation.cc:240] Negotiation successful
0929 19:06:05.682254 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260929 19:06:05.682711 25179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:05.676035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58830 (local address 127.8.158.66:32803)
0929 19:06:05.678613 (+ 2578us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:05.678617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:05.678627 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:06:05.678663 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:05.678665 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:05.678680 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:05.678731 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:06:05.679127 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:05.680141 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:05.681319 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:05.682010 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:05.682498 (+ 488us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:05.682559 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:05.682584 (+ 25us) server_negotiation.cc:299] Negotiation successful
0929 19:06:05.682637 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2468,"thread_start_us":91,"threads_started":1}
I20260929 19:06:05.683382 24974 tablet_service.cc:1467] Processing CreateTablet for tablet 2f6c4df30be3407f8bf8225c25c09a78 (DEFAULT_TABLE table=test-table [id=1dc0d5a03ad04941b7f38f74ff6c75bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:06:05.683691 24974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f6c4df30be3407f8bf8225c25c09a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:06:05.685863 25182 tablet_bootstrap.cc:492] T 2f6c4df30be3407f8bf8225c25c09a78 P 1182f2efaaae4dfbab1b8ad73a06610b: Bootstrap starting.
I20260929 19:06:05.686719 25182 tablet_bootstrap.cc:654] T 2f6c4df30be3407f8bf8225c25c09a78 P 1182f2efaaae4dfbab1b8ad73a06610b: Neither blocks nor log segments found. Creating new log.
I20260929 19:06:05.687124 25183 tablet_bootstrap.cc:492] T 2f6c4df30be3407f8bf8225c25c09a78 P 2692934c705841b5ac802c10868baf47: Bootstrap starting.
I20260929 19:06:05.687536 25182 log.cc:824] T 2f6c4df30be3407f8bf8225c25c09a78 P 1182f2efaaae4dfbab1b8ad73a06610b: Log is configured to *not* fsync() on all Append() calls
I20260929 19:06:05.687839 25183 tablet_bootstrap.cc:654] T 2f6c4df30be3407f8bf8225c25c09a78 P 2692934c705841b5ac802c10868baf47: Neither blocks nor log segments found. Creating new log.
I20260929 19:06:05.688426 25182 tablet_bootstrap.cc:492] T 2f6c4df30be3407f8bf8225c25c09a78 P 1182f2efaaae4dfbab1b8ad73a06610b: No bootstrap required, opened a new log
I20260929 19:06:05.688511 25182 ts_tablet_manager.cc:1397] T 2f6c4df30be3407f8bf8225c25c09a78 P 1182f2efaaae4dfbab1b8ad73a06610b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 19:06:05.688488 25183 log.cc:824] T 2f6c4df30be3407f8bf8225c25c09a78 P 2692934c705841b5ac802c10868baf47: Log is configured to *not* fsync() on all Append() calls
I20260929 19:06:05.689286 25183 tablet_bootstrap.cc:492] T 2f6c4df30be3407f8bf8225c25c09a78 P 2692934c705841b5ac802c10868baf47: No bootstrap required, opened a new log
I20260929 19:06:05.689364 25183 ts_tablet_manager.cc:1397] T 2f6c4df30be3407f8bf8225c25c09a78 P 2692934c705841b5ac802c10868baf47: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:06:05.690824 25183 raft_consensus.cc:348] T 2f6c4df30be3407f8bf8225c25c09a78 P 2692934c705841b5ac802c10868baf47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2692934c705841b5ac802c10868baf47" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 32803 } } peers { permanent_uuid: "1182f2efaaae4dfbab1b8ad73a06610b" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45145 } } peers { permanent_uuid: "ec0920ea49e24bbd938e0a19722d72f0" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 35205 } }
I20260929 19:06:05.690974 25183 raft_consensus.cc:374] T 2f6c4df30be3407f8bf8225c25c09a78 P 2692934c705841b5ac802c10868baf47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:06:05.691018 25183 raft_consensus.cc:729] T 2f6c4df30be3407f8bf8225c25c09a78 P 2692934c705841b5ac802c10868baf47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2692934c705841b5ac802c10868baf47, State: Initialized, Role: FOLLOWER
I20260929 19:06:05.691141 25183 consensus_queue.cc:260] T 2f6c4df30be3407f8bf8225c25c09a78 P 2692934c705841b5ac802c10868baf47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2692934c705841b5ac802c10868baf47" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 32803 } } peers { permanent_uuid: "1182f2efaaae4dfbab1b8ad73a06610b" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45145 } } peers { permanent_uuid: "ec0920ea49e24bbd938e0a19722d72f0" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 35205 } }
I20260929 19:06:05.691424 25183 ts_tablet_manager.cc:1428] T 2f6c4df30be3407f8bf8225c25c09a78 P 2692934c705841b5ac802c10868baf47: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:06:05.691434 25040 heartbeater.cc:503] Master 127.8.158.126:35519 was elected leader, sending a full tablet report...
I20260929 19:06:05.691227 25182 raft_consensus.cc:348] T 2f6c4df30be3407f8bf8225c25c09a78 P 1182f2efaaae4dfbab1b8ad73a06610b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2692934c705841b5ac802c10868baf47" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 32803 } } peers { permanent_uuid: "1182f2efaaae4dfbab1b8ad73a06610b" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45145 } } peers { permanent_uuid: "ec0920ea49e24bbd938e0a19722d72f0" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 35205 } }
I20260929 19:06:05.691514 25182 raft_consensus.cc:374] T 2f6c4df30be3407f8bf8225c25c09a78 P 1182f2efaaae4dfbab1b8ad73a06610b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:06:05.691579 25182 raft_consensus.cc:729] T 2f6c4df30be3407f8bf8225c25c09a78 P 1182f2efaaae4dfbab1b8ad73a06610b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1182f2efaaae4dfbab1b8ad73a06610b, State: Initialized, Role: FOLLOWER
I20260929 19:06:05.691712 25182 consensus_queue.cc:260] T 2f6c4df30be3407f8bf8225c25c09a78 P 1182f2efaaae4dfbab1b8ad73a06610b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2692934c705841b5ac802c10868baf47" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 32803 } } peers { permanent_uuid: "1182f2efaaae4dfbab1b8ad73a06610b" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45145 } } peers { permanent_uuid: "ec0920ea49e24bbd938e0a19722d72f0" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 35205 } }
I20260929 19:06:05.692052 25172 heartbeater.cc:503] Master 127.8.158.126:35519 was elected leader, sending a full tablet report...
I20260929 19:06:05.692055 25182 ts_tablet_manager.cc:1428] T 2f6c4df30be3407f8bf8225c25c09a78 P 1182f2efaaae4dfbab1b8ad73a06610b: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
W20260929 19:06:05.751068 25041 tablet.cc:2369] T 2f6c4df30be3407f8bf8225c25c09a78 P 2692934c705841b5ac802c10868baf47: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 19:06:05.874953 24903 tablet.cc:2369] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 19:06:05.882524 25173 tablet.cc:2369] T 2f6c4df30be3407f8bf8225c25c09a78 P 1182f2efaaae4dfbab1b8ad73a06610b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:06:05.944702 25187 raft_consensus.cc:482] T 2f6c4df30be3407f8bf8225c25c09a78 P 1182f2efaaae4dfbab1b8ad73a06610b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:06:05.944818 25187 raft_consensus.cc:504] T 2f6c4df30be3407f8bf8225c25c09a78 P 1182f2efaaae4dfbab1b8ad73a06610b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2692934c705841b5ac802c10868baf47" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 32803 } } peers { permanent_uuid: "1182f2efaaae4dfbab1b8ad73a06610b" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45145 } } peers { permanent_uuid: "ec0920ea49e24bbd938e0a19722d72f0" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 35205 } }
I20260929 19:06:05.945258 25187 leader_election.cc:288] T 2f6c4df30be3407f8bf8225c25c09a78 P 1182f2efaaae4dfbab1b8ad73a06610b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2692934c705841b5ac802c10868baf47 (127.8.158.66:32803), ec0920ea49e24bbd938e0a19722d72f0 (127.8.158.65:35205)
I20260929 19:06:05.949368 25189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:05.945549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:58727 (local address 127.8.158.65:35205)
0929 19:06:05.945662 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:05.945664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:05.945672 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:06:05.945724 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:05.945725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:05.945736 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:05.945806 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:06:05.946020 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:05.946794 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:05.948391 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:05.948982 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:05.949046 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:05.949056 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:05.949231 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 19:06:05.949294 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":59,"threads_started":1}
I20260929 19:06:05.949370 25179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:05.945527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:36301 (local address 127.8.158.66:32803)
0929 19:06:05.945577 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:05.945580 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:05.945740 (+ 160us) server_negotiation.cc:407] Connection header received
0929 19:06:05.945777 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:05.945778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:05.945789 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:05.945840 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:06:05.946019 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:05.946794 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:05.948391 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:05.948960 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:05.949041 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:05.949049 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:05.949277 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 19:06:05.949313 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 19:06:05.949405 25174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:05.945469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:35205 (local address 127.8.158.67:58727)
0929 19:06:05.945529 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:05.945541 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:05.945607 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:05.945831 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:05.945832 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:05.945835 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:06:05.945941 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:05.945944 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:05.946882 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:05.946883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:05.948291 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:05.948294 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:05.949062 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:05.949063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:05.949148 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:05.949153 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 19:06:05.949221 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 19:06:05.949273 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":29}
I20260929 19:06:05.949606 25188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:05.945471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.66:32803 (local address 127.8.158.67:36301)
0929 19:06:05.945652 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:05.945662 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:05.945736 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:05.945841 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:05.945841 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:05.945842 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:06:05.945941 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:05.945944 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:05.946882 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:05.946883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:05.948291 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:05.948294 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:05.949134 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:05.949136 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:05.949206 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:05.949210 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:06:05.949265 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 19:06:05.949294 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":144,"thread_start_us":116,"threads_started":1}
I20260929 19:06:05.949735 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f6c4df30be3407f8bf8225c25c09a78" candidate_uuid: "1182f2efaaae4dfbab1b8ad73a06610b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec0920ea49e24bbd938e0a19722d72f0" is_pre_election: true
I20260929 19:06:05.949919 24856 raft_consensus.cc:2454] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1182f2efaaae4dfbab1b8ad73a06610b in term 0.
I20260929 19:06:05.949945 24994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f6c4df30be3407f8bf8225c25c09a78" candidate_uuid: "1182f2efaaae4dfbab1b8ad73a06610b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2692934c705841b5ac802c10868baf47" is_pre_election: true
I20260929 19:06:05.950080 24994 raft_consensus.cc:2454] T 2f6c4df30be3407f8bf8225c25c09a78 P 2692934c705841b5ac802c10868baf47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1182f2efaaae4dfbab1b8ad73a06610b in term 0.
I20260929 19:06:05.950245 25059 leader_election.cc:302] T 2f6c4df30be3407f8bf8225c25c09a78 P 1182f2efaaae4dfbab1b8ad73a06610b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1182f2efaaae4dfbab1b8ad73a06610b, ec0920ea49e24bbd938e0a19722d72f0; no voters:
I20260929 19:06:05.950436 25187 raft_consensus.cc:2781] T 2f6c4df30be3407f8bf8225c25c09a78 P 1182f2efaaae4dfbab1b8ad73a06610b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:06:05.950497 25187 raft_consensus.cc:482] T 2f6c4df30be3407f8bf8225c25c09a78 P 1182f2efaaae4dfbab1b8ad73a06610b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:06:05.950515 25187 raft_consensus.cc:3037] T 2f6c4df30be3407f8bf8225c25c09a78 P 1182f2efaaae4dfbab1b8ad73a06610b [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:06:05.951148 25187 raft_consensus.cc:504] T 2f6c4df30be3407f8bf8225c25c09a78 P 1182f2efaaae4dfbab1b8ad73a06610b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2692934c705841b5ac802c10868baf47" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 32803 } } peers { permanent_uuid: "1182f2efaaae4dfbab1b8ad73a06610b" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45145 } } peers { permanent_uuid: "ec0920ea49e24bbd938e0a19722d72f0" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 35205 } }
I20260929 19:06:05.951280 25187 leader_election.cc:288] T 2f6c4df30be3407f8bf8225c25c09a78 P 1182f2efaaae4dfbab1b8ad73a06610b [CANDIDATE]: Term 1 election: Requested vote from peers 2692934c705841b5ac802c10868baf47 (127.8.158.66:32803), ec0920ea49e24bbd938e0a19722d72f0 (127.8.158.65:35205)
I20260929 19:06:05.951452 24856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f6c4df30be3407f8bf8225c25c09a78" candidate_uuid: "1182f2efaaae4dfbab1b8ad73a06610b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec0920ea49e24bbd938e0a19722d72f0"
I20260929 19:06:05.951479 24994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f6c4df30be3407f8bf8225c25c09a78" candidate_uuid: "1182f2efaaae4dfbab1b8ad73a06610b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2692934c705841b5ac802c10868baf47"
I20260929 19:06:05.951546 24856 raft_consensus.cc:3037] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:06:05.951558 24994 raft_consensus.cc:3037] T 2f6c4df30be3407f8bf8225c25c09a78 P 2692934c705841b5ac802c10868baf47 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:06:05.952210 24994 raft_consensus.cc:2454] T 2f6c4df30be3407f8bf8225c25c09a78 P 2692934c705841b5ac802c10868baf47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1182f2efaaae4dfbab1b8ad73a06610b in term 1.
I20260929 19:06:05.952330 24856 raft_consensus.cc:2454] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1182f2efaaae4dfbab1b8ad73a06610b in term 1.
I20260929 19:06:05.952426 25060 leader_election.cc:302] T 2f6c4df30be3407f8bf8225c25c09a78 P 1182f2efaaae4dfbab1b8ad73a06610b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1182f2efaaae4dfbab1b8ad73a06610b, 2692934c705841b5ac802c10868baf47; no voters:
I20260929 19:06:05.952520 25187 raft_consensus.cc:2781] T 2f6c4df30be3407f8bf8225c25c09a78 P 1182f2efaaae4dfbab1b8ad73a06610b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:06:05.952710 25187 raft_consensus.cc:686] T 2f6c4df30be3407f8bf8225c25c09a78 P 1182f2efaaae4dfbab1b8ad73a06610b [term 1 LEADER]: Becoming Leader. State: Replica: 1182f2efaaae4dfbab1b8ad73a06610b, State: Running, Role: LEADER
I20260929 19:06:05.952813 25187 consensus_queue.cc:237] T 2f6c4df30be3407f8bf8225c25c09a78 P 1182f2efaaae4dfbab1b8ad73a06610b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2692934c705841b5ac802c10868baf47" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 32803 } } peers { permanent_uuid: "1182f2efaaae4dfbab1b8ad73a06610b" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45145 } } peers { permanent_uuid: "ec0920ea49e24bbd938e0a19722d72f0" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 35205 } }
I20260929 19:06:05.953756 24713 catalog_manager.cc:5495] T 2f6c4df30be3407f8bf8225c25c09a78 P 1182f2efaaae4dfbab1b8ad73a06610b reported cstate change: term changed from 0 to 1, leader changed from <none> to 1182f2efaaae4dfbab1b8ad73a06610b (127.8.158.67). New cstate: current_term: 1 leader_uuid: "1182f2efaaae4dfbab1b8ad73a06610b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2692934c705841b5ac802c10868baf47" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 32803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1182f2efaaae4dfbab1b8ad73a06610b" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec0920ea49e24bbd938e0a19722d72f0" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 35205 } health_report { overall_health: UNKNOWN } } }
I20260929 19:06:05.966066 25180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:05.963601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35488 (local address 127.8.158.67:45145)
0929 19:06:05.963664 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:05.963666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:05.963674 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:06:05.963756 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:05.963757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:05.963772 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:05.963838 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:06:05.963936 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:05.964455 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:05.965289 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:05.965484 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:05.965542 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:05.965702 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:06:05.965854 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:06:05.965928 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:05.965982 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 19:06:05.966000 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 19:06:05.967922 24856 raft_consensus.cc:1261] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0 [term 1 FOLLOWER]: Refusing update from remote peer 1182f2efaaae4dfbab1b8ad73a06610b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:06:05.967950 24994 raft_consensus.cc:1261] T 2f6c4df30be3407f8bf8225c25c09a78 P 2692934c705841b5ac802c10868baf47 [term 1 FOLLOWER]: Refusing update from remote peer 1182f2efaaae4dfbab1b8ad73a06610b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:06:05.968283 25190 consensus_queue.cc:1035] T 2f6c4df30be3407f8bf8225c25c09a78 P 1182f2efaaae4dfbab1b8ad73a06610b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec0920ea49e24bbd938e0a19722d72f0" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 35205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:06:05.968431 25187 consensus_queue.cc:1035] T 2f6c4df30be3407f8bf8225c25c09a78 P 1182f2efaaae4dfbab1b8ad73a06610b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2692934c705841b5ac802c10868baf47" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 32803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:06:05.970548 25192 mvcc.cc:205] Tried to move back new op lower bound from 7334743105401487360 to 7334743105343418368. Current Snapshot: MvccSnapshot[applied={T|T < 7334743105401487360}]
I20260929 19:06:05.973171 24714 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:06:05.973255 24714 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:06:05.974030 24714 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0: Sending DeleteTablet for 3 replicas of tablet 2f6c4df30be3407f8bf8225c25c09a78
I20260929 19:06:05.974344 24836 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f6c4df30be3407f8bf8225c25c09a78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:06:05 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:47550
I20260929 19:06:05.974382 24974 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f6c4df30be3407f8bf8225c25c09a78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:06:05 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:58830
I20260929 19:06:05.974427 25106 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f6c4df30be3407f8bf8225c25c09a78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:06:05 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:35486
I20260929 19:06:05.974546 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 24753
I20260929 19:06:05.975286 25203 tablet_replica.cc:330] T 2f6c4df30be3407f8bf8225c25c09a78 P 2692934c705841b5ac802c10868baf47: stopping tablet replica
I20260929 19:06:05.975426 25203 raft_consensus.cc:2229] T 2f6c4df30be3407f8bf8225c25c09a78 P 2692934c705841b5ac802c10868baf47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:06:05.975566 25203 raft_consensus.cc:2258] T 2f6c4df30be3407f8bf8225c25c09a78 P 2692934c705841b5ac802c10868baf47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:06:05.976042 25203 ts_tablet_manager.cc:1905] T 2f6c4df30be3407f8bf8225c25c09a78 P 2692934c705841b5ac802c10868baf47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:06:05.977829 25203 ts_tablet_manager.cc:1918] T 2f6c4df30be3407f8bf8225c25c09a78 P 2692934c705841b5ac802c10868baf47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:06:05.977890 25203 log.cc:1196] T 2f6c4df30be3407f8bf8225c25c09a78 P 2692934c705841b5ac802c10868baf47: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-1/wal/wals/2f6c4df30be3407f8bf8225c25c09a78
I20260929 19:06:05.978160 25203 ts_tablet_manager.cc:1939] T 2f6c4df30be3407f8bf8225c25c09a78 P 2692934c705841b5ac802c10868baf47: Deleting consensus metadata
I20260929 19:06:05.978421 25204 tablet_replica.cc:330] T 2f6c4df30be3407f8bf8225c25c09a78 P 1182f2efaaae4dfbab1b8ad73a06610b: stopping tablet replica
I20260929 19:06:05.978591 25204 raft_consensus.cc:2229] T 2f6c4df30be3407f8bf8225c25c09a78 P 1182f2efaaae4dfbab1b8ad73a06610b [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:06:05.978579 24699 catalog_manager.cc:4841] TS 2692934c705841b5ac802c10868baf47 (127.8.158.66:32803): tablet 2f6c4df30be3407f8bf8225c25c09a78 (table test-table [id=1dc0d5a03ad04941b7f38f74ff6c75bf]) successfully deleted
I20260929 19:06:05.978688 25204 raft_consensus.cc:2258] T 2f6c4df30be3407f8bf8225c25c09a78 P 1182f2efaaae4dfbab1b8ad73a06610b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:06:05.979113 25204 ts_tablet_manager.cc:1905] T 2f6c4df30be3407f8bf8225c25c09a78 P 1182f2efaaae4dfbab1b8ad73a06610b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:06:05.980518 25204 ts_tablet_manager.cc:1918] T 2f6c4df30be3407f8bf8225c25c09a78 P 1182f2efaaae4dfbab1b8ad73a06610b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:06:05.980604 25204 log.cc:1196] T 2f6c4df30be3407f8bf8225c25c09a78 P 1182f2efaaae4dfbab1b8ad73a06610b: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-2/wal/wals/2f6c4df30be3407f8bf8225c25c09a78
I20260929 19:06:05.980849 25204 ts_tablet_manager.cc:1939] T 2f6c4df30be3407f8bf8225c25c09a78 P 1182f2efaaae4dfbab1b8ad73a06610b: Deleting consensus metadata
I20260929 19:06:05.981227 24698 catalog_manager.cc:4841] TS 1182f2efaaae4dfbab1b8ad73a06610b (127.8.158.67:45145): tablet 2f6c4df30be3407f8bf8225c25c09a78 (table test-table [id=1dc0d5a03ad04941b7f38f74ff6c75bf]) successfully deleted
W20260929 19:06:05.981825 24698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.158.65:35205 (error 108)
I20260929 19:06:05.981993 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 24906
I20260929 19:06:05.982962 24909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:05.982834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:35205 (local address 127.0.0.1:47560)
0929 19:06:05.982893 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:05.982938 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.65:35205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 19:06:05.983143 24698 catalog_manager.cc:4584] TS ec0920ea49e24bbd938e0a19722d72f0 (127.8.158.65:35205): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2f6c4df30be3407f8bf8225c25c09a78: Network error: Client connection negotiation failed: client connection to 127.8.158.65:35205: connect: Connection refused (error 111)
I20260929 19:06:05.987941 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 25043
I20260929 19:06:05.993276 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 24681
I20260929 19:06:25.998535 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:35519
--webserver_interface=127.8.158.126
--webserver_port=44353
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:43293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:35519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:06:26.081159 25206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:06:26.081338 25206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:06:26.081360 25206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:06:26.082422 25206 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 19:06:26.082453 25206 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 19:06:26.082465 25206 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 19:06:26.082477 25206 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:06:26.082489 25206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:06:26.082502 25206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:06:26.082515 25206 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:06:26.082526 25206 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:06:26.083951 25206 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:43293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:35519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:35519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.158.126
--webserver_port=44353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:06:26.084121 25206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:06:26.084275 25206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:06:26.089337 25214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:26.089344 25211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:26.089414 25206 server_base.cc:1034] running on GCE node
W20260929 19:06:26.089347 25212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:26.089772 25206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:06:26.089998 25206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:06:26.091143 25206 hybrid_clock.cc:648] HybridClock initialized: now 1790708786091125 us; error 38 us; skew 500 ppm
Sep 29 19:06:26 dist-test-slave-nmsz krb5kdc[24138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708786, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:06:26.093566 25206 init.cc:377] Logged in from keytab as kudu/127.8.158.126@KRBTEST.COM (short username kudu)
I20260929 19:06:26.093957 25206 webserver.cc:460] Webserver started at http://127.8.158.126:44353/ using document root <none> and password file <none>
I20260929 19:06:26.094183 25206 fs_manager.cc:359] Metadata directory not provided
I20260929 19:06:26.094226 25206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:06:26.095551 25206 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:06:26.096247 25221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:26.096418 25206 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:06:26.096525 25206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "443331b8c71b4bbea9b6cc9ff9bca0b0"
format_stamp: "Formatted at 2026-09-29 19:05:44 on dist-test-slave-nmsz"
server_key: "45f265def8124e8902fc2312f7e7e437"
server_key_iv: "8f3997a84fbf126c4b34f1fab07770fd"
server_key_version: "encryptionkey@0"
I20260929 19:06:26.096994 25206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:06:26.116036 25206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:06:26.118294 25206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:06:26.118458 25206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:06:26.122169 25206 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.126:35519
I20260929 19:06:26.122256 25273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.126:35519 every 8 connection(s)
I20260929 19:06:26.122606 25206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:06:26.123353 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 25206
I20260929 19:06:26.123656 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.65:35205
--local_ip_for_outbound_sockets=127.8.158.65
--tserver_master_addrs=127.8.158.126:35519
--webserver_port=36399
--webserver_interface=127.8.158.65
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab
--principal=kudu/127.8.158.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:43293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:06:26.124190 25274 sys_catalog.cc:263] Verifying existing consensus state
I20260929 19:06:26.124830 25274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0: Bootstrap starting.
I20260929 19:06:26.133199 25274 log.cc:824] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0: Log is configured to *not* fsync() on all Append() calls
I20260929 19:06:26.135423 25274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 19:06:26.135717 25274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0: Bootstrap complete.
I20260929 19:06:26.137969 25274 raft_consensus.cc:348] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443331b8c71b4bbea9b6cc9ff9bca0b0" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35519 } }
I20260929 19:06:26.138209 25274 raft_consensus.cc:729] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443331b8c71b4bbea9b6cc9ff9bca0b0, State: Initialized, Role: FOLLOWER
I20260929 19:06:26.138430 25274 consensus_queue.cc:260] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443331b8c71b4bbea9b6cc9ff9bca0b0" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35519 } }
I20260929 19:06:26.138530 25274 raft_consensus.cc:388] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:06:26.138583 25274 raft_consensus.cc:482] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:06:26.138631 25274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [term 2 FOLLOWER]: Advancing to term 3
I20260929 19:06:26.139552 25274 raft_consensus.cc:504] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443331b8c71b4bbea9b6cc9ff9bca0b0" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35519 } }
I20260929 19:06:26.139669 25274 leader_election.cc:302] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 443331b8c71b4bbea9b6cc9ff9bca0b0; no voters:
I20260929 19:06:26.139845 25274 leader_election.cc:288] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 19:06:26.139909 25278 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 19:06:26.140067 25278 raft_consensus.cc:686] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [term 3 LEADER]: Becoming Leader. State: Replica: 443331b8c71b4bbea9b6cc9ff9bca0b0, State: Running, Role: LEADER
I20260929 19:06:26.140112 25274 sys_catalog.cc:564] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:06:26.140234 25278 consensus_queue.cc:237] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443331b8c71b4bbea9b6cc9ff9bca0b0" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35519 } }
I20260929 19:06:26.140527 25279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "443331b8c71b4bbea9b6cc9ff9bca0b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443331b8c71b4bbea9b6cc9ff9bca0b0" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35519 } } }
I20260929 19:06:26.140538 25280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 443331b8c71b4bbea9b6cc9ff9bca0b0. Latest consensus state: current_term: 3 leader_uuid: "443331b8c71b4bbea9b6cc9ff9bca0b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443331b8c71b4bbea9b6cc9ff9bca0b0" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35519 } } }
I20260929 19:06:26.140676 25279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [sys.catalog]: This master's current role is: LEADER
I20260929 19:06:26.140691 25280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0 [sys.catalog]: This master's current role is: LEADER
I20260929 19:06:26.141304 25285 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:06:26.142084 25285 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:06:26.142216 25285 catalog_manager.cc:1259] Loaded cluster ID: 7beb21957b384c0b96202505a0ea581c
I20260929 19:06:26.142484 25285 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:06:26.143707 25285 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:06:26.143872 25285 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0: Loaded TSK: 0
I20260929 19:06:26.143918 25285 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 443331b8c71b4bbea9b6cc9ff9bca0b0: Loaded TSK: 1
F20260929 19:06:26.144152 25285 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe465a879ad google::LogMessage::Fail() at ??:0
@ 0x7fe465a8a88a google::LogMessage::SendToLog() at ??:0
@ 0x7fe465a874de google::LogMessage::Flush() at ??:0
@ 0x7fe465a88a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe4681f4104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fe4681f1d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fe46d9e11cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fe46d9e148e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fe46d9e1594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fe46d9a7c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fe46d9b1ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe4679550d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe46794a954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe4687ae6db start_thread at ??:0
@ 0x7fe46458561f clone at ??:0
W20260929 19:06:26.220342 25276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:06:26.220556 25276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:06:26.220594 25276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:06:26.221728 25276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:06:26.221796 25276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:06:26.221840 25276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.65
I20260929 19:06:26.223366 25276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:43293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab
--principal=kudu/127.8.158.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.65:35205
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.158.65
--webserver_port=36399
--tserver_master_addrs=127.8.158.126:35519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.65
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:06:26.223619 25276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:06:26.223839 25276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:06:26.229832 25316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:26.229871 25318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:26.230160 25276 server_base.cc:1034] running on GCE node
W20260929 19:06:26.229890 25315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:26.230459 25276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:06:26.230720 25276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:06:26.231870 25276 hybrid_clock.cc:648] HybridClock initialized: now 1790708786231855 us; error 32 us; skew 500 ppm
Sep 29 19:06:26 dist-test-slave-nmsz krb5kdc[24138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708786, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:06:26.234247 25276 init.cc:377] Logged in from keytab as kudu/127.8.158.65@KRBTEST.COM (short username kudu)
I20260929 19:06:26.234733 25276 webserver.cc:460] Webserver started at http://127.8.158.65:36399/ using document root <none> and password file <none>
I20260929 19:06:26.234957 25276 fs_manager.cc:359] Metadata directory not provided
I20260929 19:06:26.235025 25276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:06:26.236362 25276 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:06:26.237118 25325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:26.237272 25276 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:06:26.237365 25276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-0/wal
uuid: "ec0920ea49e24bbd938e0a19722d72f0"
format_stamp: "Formatted at 2026-09-29 19:05:44 on dist-test-slave-nmsz"
server_key: "bc1bdfe877c751866b10d0bd553cbbb5"
server_key_iv: "70443c5e45a036b800f7f0c61e068ddb"
server_key_version: "encryptionkey@0"
I20260929 19:06:26.237685 25276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:06:26.249886 25276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:06:26.252168 25276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:06:26.252324 25276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:06:26.252542 25276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:06:26.252942 25332 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 19:06:26.254858 25276 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 19:06:26.254922 25276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260929 19:06:26.254962 25276 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 19:06:26.255574 25276 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 19:06:26.255635 25276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260929 19:06:26.255697 25332 tablet_bootstrap.cc:492] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0: Bootstrap starting.
I20260929 19:06:26.262962 25276 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.65:35205
I20260929 19:06:26.263083 25439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.65:35205 every 8 connection(s)
I20260929 19:06:26.263363 25276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
I20260929 19:06:26.266644 25332 log.cc:824] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0: Log is configured to *not* fsync() on all Append() calls
I20260929 19:06:26.268641 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 25276
I20260929 19:06:26.268900 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.66:32803
--local_ip_for_outbound_sockets=127.8.158.66
--tserver_master_addrs=127.8.158.126:35519
--webserver_port=43479
--webserver_interface=127.8.158.66
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab
--principal=kudu/127.8.158.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:43293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:06:26.269117 25332 tablet_bootstrap.cc:492] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 19:06:26.269450 25332 tablet_bootstrap.cc:492] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0: Bootstrap complete.
I20260929 19:06:26.270691 25332 ts_tablet_manager.cc:1397] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.010s
I20260929 19:06:26.272060 25332 raft_consensus.cc:348] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2692934c705841b5ac802c10868baf47" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 32803 } } peers { permanent_uuid: "1182f2efaaae4dfbab1b8ad73a06610b" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45145 } } peers { permanent_uuid: "ec0920ea49e24bbd938e0a19722d72f0" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 35205 } }
I20260929 19:06:26.272369 25332 raft_consensus.cc:729] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec0920ea49e24bbd938e0a19722d72f0, State: Initialized, Role: FOLLOWER
I20260929 19:06:26.272500 25332 consensus_queue.cc:260] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2692934c705841b5ac802c10868baf47" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 32803 } } peers { permanent_uuid: "1182f2efaaae4dfbab1b8ad73a06610b" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45145 } } peers { permanent_uuid: "ec0920ea49e24bbd938e0a19722d72f0" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 35205 } }
I20260929 19:06:26.272735 25332 ts_tablet_manager.cc:1428] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20260929 19:06:26.340862 25442 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:06:26.264039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.65:52285)
0929 19:06:26.264206 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:26.264248 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:26.264704 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:26.340647 (+ 75943us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":101,"threads_started":1}
W20260929 19:06:26.341135 25440 heartbeater.cc:644] Failed to heartbeat to 127.8.158.126:35519 (0 consecutive failures): Network error: Failed to ping master at 127.8.158.126:35519: Client connection negotiation failed: client connection to 127.8.158.126:35519: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 19:06:26.341506 25442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:26.341372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.65:40325)
0929 19:06:26.341431 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:26.341479 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260929 19:06:26.341835 25442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:26.341734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.65:33703)
0929 19:06:26.341785 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:26.341813 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 19:06:26.341944 25440 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 19:06:26.349784 25444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:06:26.349956 25444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:06:26.349977 25444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:06:26.351159 25444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:06:26.351200 25444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:06:26.351226 25444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.66
I20260929 19:06:26.352623 25444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:43293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab
--principal=kudu/127.8.158.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.66:32803
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.158.66
--webserver_port=43479
--tserver_master_addrs=127.8.158.126:35519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.66
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:06:26.352862 25444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:06:26.353080 25444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:06:26.357807 25450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:26.357836 25453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:26.357837 25451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:26.358047 25444 server_base.cc:1034] running on GCE node
I20260929 19:06:26.358265 25444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:06:26.358527 25444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:06:26.359697 25444 hybrid_clock.cc:648] HybridClock initialized: now 1790708786359683 us; error 27 us; skew 500 ppm
Sep 29 19:06:26 dist-test-slave-nmsz krb5kdc[24138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708786, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:06:26.362059 25444 init.cc:377] Logged in from keytab as kudu/127.8.158.66@KRBTEST.COM (short username kudu)
I20260929 19:06:26.362551 25444 webserver.cc:460] Webserver started at http://127.8.158.66:43479/ using document root <none> and password file <none>
I20260929 19:06:26.362778 25444 fs_manager.cc:359] Metadata directory not provided
I20260929 19:06:26.362845 25444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:06:26.364143 25444 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:06:26.364732 25460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:26.364939 25444 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:06:26.365039 25444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-1/wal
uuid: "2692934c705841b5ac802c10868baf47"
format_stamp: "Formatted at 2026-09-29 19:05:44 on dist-test-slave-nmsz"
server_key: "37e781166689b43d535f360e36283177"
server_key_iv: "acbf5dab4566e76de08ad740536bf57c"
server_key_version: "encryptionkey@0"
I20260929 19:06:26.365337 25444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:06:26.375533 25444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:06:26.377616 25444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:06:26.377785 25444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:06:26.377997 25444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:06:26.378279 25444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:06:26.378350 25444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:26.378391 25444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:06:26.378420 25444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:26.384249 25444 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.66:32803
I20260929 19:06:26.384299 25573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.66:32803 every 8 connection(s)
I20260929 19:06:26.384626 25444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
I20260929 19:06:26.385264 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 25444
I20260929 19:06:26.385574 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.67:45145
--local_ip_for_outbound_sockets=127.8.158.67
--tserver_master_addrs=127.8.158.126:35519
--webserver_port=36083
--webserver_interface=127.8.158.67
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab
--principal=kudu/127.8.158.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:43293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:06:26.385757 25576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:26.385304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.66:55401)
0929 19:06:26.385489 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:26.385569 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":100,"threads_started":1}
W20260929 19:06:26.386122 25574 heartbeater.cc:644] Failed to heartbeat to 127.8.158.126:35519 (0 consecutive failures): Network error: Failed to ping master at 127.8.158.126:35519: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
I20260929 19:06:26.386418 25576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:26.386304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.66:43583)
0929 19:06:26.386373 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:26.386397 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 19:06:26.386803 25576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:26.386664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.66:44171)
0929 19:06:26.386754 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:26.386784 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260929 19:06:26.387058 25574 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 19:06:26.465309 25577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:06:26.465487 25577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:06:26.465510 25577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:06:26.466655 25577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:06:26.466696 25577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:06:26.466722 25577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.67
I20260929 19:06:26.468178 25577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:43293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab
--principal=kudu/127.8.158.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.67:45145
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.158.67
--webserver_port=36083
--tserver_master_addrs=127.8.158.126:35519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.67
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:06:26.468401 25577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:06:26.468626 25577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:06:26.473752 25582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:26.473760 25583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:26.473769 25585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:26.474017 25577 server_base.cc:1034] running on GCE node
I20260929 19:06:26.474277 25577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:06:26.474541 25577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:06:26.475699 25577 hybrid_clock.cc:648] HybridClock initialized: now 1790708786475694 us; error 40 us; skew 500 ppm
Sep 29 19:06:26 dist-test-slave-nmsz krb5kdc[24138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708786, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:06:26.478041 25577 init.cc:377] Logged in from keytab as kudu/127.8.158.67@KRBTEST.COM (short username kudu)
I20260929 19:06:26.478555 25577 webserver.cc:460] Webserver started at http://127.8.158.67:36083/ using document root <none> and password file <none>
I20260929 19:06:26.478793 25577 fs_manager.cc:359] Metadata directory not provided
I20260929 19:06:26.478860 25577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:06:26.480185 25577 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:06:26.480808 25592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:26.480988 25577 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:06:26.481082 25577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-2/wal
uuid: "1182f2efaaae4dfbab1b8ad73a06610b"
format_stamp: "Formatted at 2026-09-29 19:05:44 on dist-test-slave-nmsz"
server_key: "3da672f1d83b864f3adea6c98b9cf023"
server_key_iv: "2857cbb5cec791b823d0043eec7440d1"
server_key_version: "encryptionkey@0"
I20260929 19:06:26.481396 25577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:06:26.504724 25577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:06:26.506767 25577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:06:26.506927 25577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:06:26.507164 25577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:06:26.507479 25577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:06:26.507535 25577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:26.507570 25577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:06:26.507596 25577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:26.513420 25577 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.67:45145
I20260929 19:06:26.513473 25705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.67:45145 every 8 connection(s)
I20260929 19:06:26.513777 25577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
I20260929 19:06:26.514843 25708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:26.514446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.67:34367)
0929 19:06:26.514590 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:26.514679 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":81,"threads_started":1}
W20260929 19:06:26.515226 25706 heartbeater.cc:644] Failed to heartbeat to 127.8.158.126:35519 (0 consecutive failures): Network error: Failed to ping master at 127.8.158.126:35519: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
I20260929 19:06:26.515511 25708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:26.515410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.67:37967)
0929 19:06:26.515464 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:26.515494 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 19:06:26.515771 25708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:26.515685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.67:58479)
0929 19:06:26.515731 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:26.515753 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260929 19:06:26.515877 25706 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 19:06:26.520691 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 25577
I20260929 19:06:27.342540 25710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:27.342260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.65:53729)
0929 19:06:27.342437 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:27.342477 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":95,"threads_started":1}
I20260929 19:06:27.387598 25711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:27.387396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.66:45885)
0929 19:06:27.387521 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:27.387554 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":55,"threads_started":1}
I20260929 19:06:27.516319 25712 raft_consensus.cc:482] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:06:27.516494 25713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:27.516255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.67:57561)
0929 19:06:27.516384 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:27.516420 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
I20260929 19:06:27.516503 25712 raft_consensus.cc:504] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2692934c705841b5ac802c10868baf47" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 32803 } } peers { permanent_uuid: "1182f2efaaae4dfbab1b8ad73a06610b" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45145 } } peers { permanent_uuid: "ec0920ea49e24bbd938e0a19722d72f0" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 35205 } }
I20260929 19:06:27.516875 25712 leader_election.cc:288] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2692934c705841b5ac802c10868baf47 (127.8.158.66:32803), 1182f2efaaae4dfbab1b8ad73a06610b (127.8.158.67:45145)
Sep 29 19:06:27 dist-test-slave-nmsz krb5kdc[24138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708786, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for kudu/127.8.158.67@KRBTEST.COM
Sep 29 19:06:27 dist-test-slave-nmsz krb5kdc[24138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708786, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for kudu/127.8.158.66@KRBTEST.COM
I20260929 19:06:27.529446 25715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:27.517142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:40867 (local address 127.8.158.66:32803)
0929 19:06:27.517274 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:27.517281 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:27.517406 (+ 125us) server_negotiation.cc:407] Connection header received
0929 19:06:27.517918 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:27.517921 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:27.517973 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:27.518082 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:27.519084 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:27.520473 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:27.521997 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:27.522631 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:27.522747 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:27.525898 (+ 3151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:27.525901 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:27.525902 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:06:27.525919 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:27.527830 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:27.528242 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:27.528244 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:27.528244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:27.528358 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:27.528648 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:27.528648 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:27.528649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:27.528848 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:27.528946 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:27.529212 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 19:06:27.529333 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":49,"threads_started":1}
I20260929 19:06:27.529435 25710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:27.516994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.67:45145 (local address 127.8.158.65:60125)
0929 19:06:27.517065 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:27.517076 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:27.517138 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:27.518151 (+ 1013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:27.518154 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:27.518687 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:06:27.518950 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:27.518957 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:27.520623 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:27.520623 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:27.521888 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:27.521891 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:27.522738 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:27.522740 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:27.522880 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:27.523530 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:06:27.523537 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:06:27.525416 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:06:27.527702 (+ 2286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:06:27.527709 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:06:27.527711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:06:27.528039 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:06:27.528330 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:06:27.528331 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:06:27.528331 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:06:27.528458 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:06:27.528910 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:06:27.528916 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:06:27.529045 (+ 129us) client_negotiation.cc:769] Sending connection context
0929 19:06:27.529150 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 19:06:27.529332 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":89}
I20260929 19:06:27.529475 25714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:27.517105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:60125 (local address 127.8.158.67:45145)
0929 19:06:27.517240 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:27.517250 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:27.517285 (+ 35us) server_negotiation.cc:407] Connection header received
0929 19:06:27.517827 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:27.517830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:27.517883 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:27.518036 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:27.519083 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:27.520477 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:27.521997 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:27.522631 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:27.522749 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:27.525585 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:27.525589 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:27.525590 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:06:27.525609 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:27.527621 (+ 2012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:27.528143 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:27.528146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:27.528147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:27.528255 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:27.528546 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:27.528547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:27.528547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:27.528834 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:27.528898 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:27.529209 (+ 311us) server_negotiation.cc:299] Negotiation successful
0929 19:06:27.529359 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
I20260929 19:06:27.529567 25716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:27.516994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.66:32803 (local address 127.8.158.65:40867)
0929 19:06:27.517330 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:27.517339 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:27.517392 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:27.518151 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:27.518154 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:27.518689 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:06:27.518950 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:27.518957 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:27.520610 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:27.520611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:27.521888 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:27.521891 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:27.522740 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:27.522741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:27.522880 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:27.523525 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:06:27.523537 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:06:27.525795 (+ 2258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:06:27.527916 (+ 2121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:06:27.527917 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:06:27.527917 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:06:27.528165 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:06:27.528471 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:06:27.528472 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:06:27.528472 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:06:27.528574 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:06:27.528949 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:06:27.528950 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:06:27.529049 (+ 99us) client_negotiation.cc:769] Sending connection context
0929 19:06:27.529148 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 19:06:27.529433 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":416,"thread_start_us":51,"threads_started":1}
W20260929 19:06:27.530229 25326 leader_election.cc:341] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1182f2efaaae4dfbab1b8ad73a06610b (127.8.158.67:45145): Not found: Tablet not found: 2f6c4df30be3407f8bf8225c25c09a78
W20260929 19:06:27.530784 25327 leader_election.cc:341] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2692934c705841b5ac802c10868baf47 (127.8.158.66:32803): Not found: Tablet not found: 2f6c4df30be3407f8bf8225c25c09a78
I20260929 19:06:27.530848 25327 leader_election.cc:302] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec0920ea49e24bbd938e0a19722d72f0; no voters: 1182f2efaaae4dfbab1b8ad73a06610b, 2692934c705841b5ac802c10868baf47
I20260929 19:06:27.530952 25712 raft_consensus.cc:2726] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 19:06:28.343247 25717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:28.343015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.65:48355)
0929 19:06:28.343158 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:28.343204 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":66,"threads_started":1}
I20260929 19:06:28.388315 25718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:28.388093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.66:49003)
0929 19:06:28.388247 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:28.388281 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":63,"threads_started":1}
I20260929 19:06:28.517306 25719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:28.517093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.67:58283)
0929 19:06:28.517224 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:28.517260 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":58,"threads_started":1}
I20260929 19:06:29.343954 25720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:29.343742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.65:52295)
0929 19:06:29.343871 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:29.343908 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":62,"threads_started":1}
I20260929 19:06:29.389014 25721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:29.388755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.66:56967)
0929 19:06:29.388931 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:29.388975 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":84,"threads_started":1}
I20260929 19:06:29.420353 25722 raft_consensus.cc:482] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:06:29.420511 25722 raft_consensus.cc:504] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2692934c705841b5ac802c10868baf47" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 32803 } } peers { permanent_uuid: "1182f2efaaae4dfbab1b8ad73a06610b" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45145 } } peers { permanent_uuid: "ec0920ea49e24bbd938e0a19722d72f0" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 35205 } }
I20260929 19:06:29.420866 25722 leader_election.cc:288] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2692934c705841b5ac802c10868baf47 (127.8.158.66:32803), 1182f2efaaae4dfbab1b8ad73a06610b (127.8.158.67:45145)
W20260929 19:06:29.421284 25326 leader_election.cc:341] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1182f2efaaae4dfbab1b8ad73a06610b (127.8.158.67:45145): Not found: Tablet not found: 2f6c4df30be3407f8bf8225c25c09a78
W20260929 19:06:29.421365 25327 leader_election.cc:341] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2692934c705841b5ac802c10868baf47 (127.8.158.66:32803): Not found: Tablet not found: 2f6c4df30be3407f8bf8225c25c09a78
I20260929 19:06:29.421479 25327 leader_election.cc:302] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec0920ea49e24bbd938e0a19722d72f0; no voters: 1182f2efaaae4dfbab1b8ad73a06610b, 2692934c705841b5ac802c10868baf47
I20260929 19:06:29.421556 25722 raft_consensus.cc:2726] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 19:06:29.517973 25723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:29.517766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.67:56355)
0929 19:06:29.517891 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:29.517935 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":56,"threads_started":1}
I20260929 19:06:30.344659 25724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:30.344430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.65:43525)
0929 19:06:30.344576 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:30.344611 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":62,"threads_started":1}
I20260929 19:06:30.389833 25725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:30.389568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.66:34429)
0929 19:06:30.389750 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:30.389790 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":69,"threads_started":1}
I20260929 19:06:30.518733 25726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:30.518512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.67:36225)
0929 19:06:30.518651 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:30.518689 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":60,"threads_started":1}
I20260929 19:06:31.345467 25727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:31.345221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.65:54657)
0929 19:06:31.345378 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:31.345418 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":75,"threads_started":1}
I20260929 19:06:31.390547 25728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:31.390280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.66:43263)
0929 19:06:31.390470 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:31.390507 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":121,"threads_started":1}
I20260929 19:06:31.519428 25729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:31.519240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.67:41277)
0929 19:06:31.519355 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:31.519389 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":56,"threads_started":1}
I20260929 19:06:32.346184 25730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:32.345943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.65:51831)
0929 19:06:32.346092 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:32.346135 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":76,"threads_started":1}
I20260929 19:06:32.391196 25731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:32.391000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.66:42281)
0929 19:06:32.391123 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:32.391156 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":60,"threads_started":1}
I20260929 19:06:32.520100 25732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:32.519883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.67:57729)
0929 19:06:32.520008 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:32.520051 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":71,"threads_started":1}
I20260929 19:06:32.602270 25733 raft_consensus.cc:482] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:06:32.602456 25733 raft_consensus.cc:504] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2692934c705841b5ac802c10868baf47" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 32803 } } peers { permanent_uuid: "1182f2efaaae4dfbab1b8ad73a06610b" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45145 } } peers { permanent_uuid: "ec0920ea49e24bbd938e0a19722d72f0" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 35205 } }
I20260929 19:06:32.602624 25733 leader_election.cc:288] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2692934c705841b5ac802c10868baf47 (127.8.158.66:32803), 1182f2efaaae4dfbab1b8ad73a06610b (127.8.158.67:45145)
W20260929 19:06:32.602985 25327 leader_election.cc:341] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2692934c705841b5ac802c10868baf47 (127.8.158.66:32803): Not found: Tablet not found: 2f6c4df30be3407f8bf8225c25c09a78
W20260929 19:06:32.603045 25326 leader_election.cc:341] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1182f2efaaae4dfbab1b8ad73a06610b (127.8.158.67:45145): Not found: Tablet not found: 2f6c4df30be3407f8bf8225c25c09a78
I20260929 19:06:32.603112 25326 leader_election.cc:302] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec0920ea49e24bbd938e0a19722d72f0; no voters: 1182f2efaaae4dfbab1b8ad73a06610b, 2692934c705841b5ac802c10868baf47
I20260929 19:06:32.603209 25733 raft_consensus.cc:2726] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 19:06:33.346975 25734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:33.346747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.65:32899)
0929 19:06:33.346894 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:33.346930 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":65,"threads_started":1}
I20260929 19:06:33.391851 25735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:33.391637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.66:60013)
0929 19:06:33.391777 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:33.391812 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":62,"threads_started":1}
I20260929 19:06:33.520702 25736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:33.520512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.67:53933)
0929 19:06:33.520631 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:33.520665 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":58,"threads_started":1}
I20260929 19:06:34.347687 25737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:34.347462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.65:49391)
0929 19:06:34.347606 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:34.347642 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":62,"threads_started":1}
I20260929 19:06:34.392522 25738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:34.392324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.66:45127)
0929 19:06:34.392450 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:34.392484 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":58,"threads_started":1}
I20260929 19:06:34.521348 25739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:34.521157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.67:35279)
0929 19:06:34.521280 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:34.521312 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":71,"threads_started":1}
I20260929 19:06:35.348419 25740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:35.348191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.65:40311)
0929 19:06:35.348332 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:35.348368 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":62,"threads_started":1}
I20260929 19:06:35.393271 25741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:35.393083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.66:52997)
0929 19:06:35.393196 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:35.393228 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":60,"threads_started":1}
I20260929 19:06:35.522121 25742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:35.521878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.67:51317)
0929 19:06:35.522034 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:35.522072 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":107,"threads_started":1}
I20260929 19:06:35.536250 25743 raft_consensus.cc:482] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:06:35.536373 25743 raft_consensus.cc:504] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2692934c705841b5ac802c10868baf47" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 32803 } } peers { permanent_uuid: "1182f2efaaae4dfbab1b8ad73a06610b" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45145 } } peers { permanent_uuid: "ec0920ea49e24bbd938e0a19722d72f0" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 35205 } }
I20260929 19:06:35.536604 25743 leader_election.cc:288] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2692934c705841b5ac802c10868baf47 (127.8.158.66:32803), 1182f2efaaae4dfbab1b8ad73a06610b (127.8.158.67:45145)
W20260929 19:06:35.537021 25327 leader_election.cc:341] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2692934c705841b5ac802c10868baf47 (127.8.158.66:32803): Not found: Tablet not found: 2f6c4df30be3407f8bf8225c25c09a78
W20260929 19:06:35.537133 25326 leader_election.cc:341] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1182f2efaaae4dfbab1b8ad73a06610b (127.8.158.67:45145): Not found: Tablet not found: 2f6c4df30be3407f8bf8225c25c09a78
I20260929 19:06:35.537220 25326 leader_election.cc:302] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec0920ea49e24bbd938e0a19722d72f0; no voters: 1182f2efaaae4dfbab1b8ad73a06610b, 2692934c705841b5ac802c10868baf47
I20260929 19:06:35.537341 25743 raft_consensus.cc:2726] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 19:06:36.349220 25744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:36.348993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.65:37859)
0929 19:06:36.349130 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:36.349176 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":60,"threads_started":1}
I20260929 19:06:36.393887 25745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:36.393703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.66:48291)
0929 19:06:36.393816 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:36.393846 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":57,"threads_started":1}
I20260929 19:06:36.522866 25746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:36.522655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.67:54921)
0929 19:06:36.522788 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:36.522824 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":61,"threads_started":1}
I20260929 19:06:37.349965 25747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:37.349727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.65:49249)
0929 19:06:37.349882 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:37.349919 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":58,"threads_started":1}
I20260929 19:06:37.394555 25748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:37.394359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.66:41199)
0929 19:06:37.394486 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:37.394516 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":90,"threads_started":1}
I20260929 19:06:37.523619 25749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:37.523422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.67:36737)
0929 19:06:37.523547 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:37.523581 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":70,"threads_started":1}
I20260929 19:06:37.837463 25750 raft_consensus.cc:482] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:06:37.837549 25750 raft_consensus.cc:504] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2692934c705841b5ac802c10868baf47" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 32803 } } peers { permanent_uuid: "1182f2efaaae4dfbab1b8ad73a06610b" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45145 } } peers { permanent_uuid: "ec0920ea49e24bbd938e0a19722d72f0" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 35205 } }
I20260929 19:06:37.837746 25750 leader_election.cc:288] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2692934c705841b5ac802c10868baf47 (127.8.158.66:32803), 1182f2efaaae4dfbab1b8ad73a06610b (127.8.158.67:45145)
W20260929 19:06:37.838158 25327 leader_election.cc:341] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2692934c705841b5ac802c10868baf47 (127.8.158.66:32803): Not found: Tablet not found: 2f6c4df30be3407f8bf8225c25c09a78
W20260929 19:06:37.838215 25326 leader_election.cc:341] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1182f2efaaae4dfbab1b8ad73a06610b (127.8.158.67:45145): Not found: Tablet not found: 2f6c4df30be3407f8bf8225c25c09a78
I20260929 19:06:37.838243 25327 leader_election.cc:302] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec0920ea49e24bbd938e0a19722d72f0; no voters: 1182f2efaaae4dfbab1b8ad73a06610b, 2692934c705841b5ac802c10868baf47
I20260929 19:06:37.838348 25750 raft_consensus.cc:2726] T 2f6c4df30be3407f8bf8225c25c09a78 P ec0920ea49e24bbd938e0a19722d72f0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 19:06:38.350713 25751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:38.350487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.65:57509)
0929 19:06:38.350618 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:38.350658 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":71,"threads_started":1}
I20260929 19:06:38.395169 25752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:38.394982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.66:54149)
0929 19:06:38.395096 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:38.395128 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":56,"threads_started":1}
I20260929 19:06:38.524389 25753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:38.524194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.67:60879)
0929 19:06:38.524316 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:38.524350 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":56,"threads_started":1}
I20260929 19:06:39.351420 25754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:39.351192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.65:58997)
0929 19:06:39.351336 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:39.351373 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":75,"threads_started":1}
I20260929 19:06:39.395921 25755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:39.395690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.66:46223)
0929 19:06:39.395828 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:39.395879 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":72,"threads_started":1}
I20260929 19:06:39.525131 25756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:39.524934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.67:57929)
0929 19:06:39.525059 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:39.525092 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":69,"threads_started":1}
I20260929 19:06:40.352131 25757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:40.351875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.65:43569)
0929 19:06:40.352050 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:40.352086 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":95,"threads_started":1}
I20260929 19:06:40.396610 25758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:40.396383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.66:60099)
0929 19:06:40.396512 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:40.396555 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":62,"threads_started":1}
I20260929 19:06:40.525724 25759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:40.525522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.67:51625)
0929 19:06:40.525632 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:40.525683 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":54,"threads_started":1}
I20260929 19:06:41.352897 25760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:41.352670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.65:45883)
0929 19:06:41.352816 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:41.352851 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":64,"threads_started":1}
I20260929 19:06:41.397276 25761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:41.397093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.66:60821)
0929 19:06:41.397206 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:41.397238 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":57,"threads_started":1}
I20260929 19:06:41.522583 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 25276
I20260929 19:06:41.526461 25762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:41.526253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35519 (local address 127.8.158.67:49919)
0929 19:06:41.526396 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:41.526425 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.126:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":94,"threads_started":1}
I20260929 19:06:41.527320 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 25444
I20260929 19:06:41.531941 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 25577
2026-09-29T19:06:41Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58684 ms)
[----------] 30 tests from SecurityITest (227803 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:06:41 dist-test-slave-nmsz krb5kdc[25765](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:06:41 dist-test-slave-nmsz krb5kdc[25765](info): set up 2 sockets
Sep 29 19:06:41 dist-test-slave-nmsz krb5kdc[25765](info): commencing operation
krb5kdc: starting...
W20260929 19:06:42.578595 8825 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Sep 29 19:06:42 dist-test-slave-nmsz krb5kdc[25765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:06:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:06:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.158.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
I20260929 19:06:42.693009 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:42189
--webserver_interface=127.8.158.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:41705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:42189
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:06:42.770627 25778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:06:42.770816 25778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:06:42.770839 25778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:06:42.771883 25778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:06:42.771914 25778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:06:42.771927 25778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:06:42.771939 25778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:06:42.771951 25778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:06:42.773288 25778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:41705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:42189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:42189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.158.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:06:42.773459 25778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:06:42.773615 25778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:06:42.778083 25786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:42.778117 25783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:42.778163 25784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:42.778367 25778 server_base.cc:1034] running on GCE node
I20260929 19:06:42.778590 25778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:06:42.778801 25778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:06:42.779935 25778 hybrid_clock.cc:648] HybridClock initialized: now 1790708802779926 us; error 27 us; skew 500 ppm
Sep 29 19:06:42 dist-test-slave-nmsz krb5kdc[25765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708802, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:06:42.782632 25778 init.cc:377] Logged in from keytab as kudu/127.8.158.126@KRBTEST.COM (short username kudu)
I20260929 19:06:42.783042 25778 webserver.cc:460] Webserver started at http://127.8.158.126:39963/ using document root <none> and password file <none>
I20260929 19:06:42.783272 25778 fs_manager.cc:359] Metadata directory not provided
I20260929 19:06:42.783339 25778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:06:42.783470 25778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:06:42.784462 25778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "42e37ba5a9134c168ae58e2676188068"
format_stamp: "Formatted at 2026-09-29 19:06:42 on dist-test-slave-nmsz"
server_key: "bbb2b07ce197963cd02d676ab03262f2"
server_key_iv: "e2155c868575955bd8a69302df37cb43"
server_key_version: "encryptionkey@0"
I20260929 19:06:42.784809 25778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "42e37ba5a9134c168ae58e2676188068"
format_stamp: "Formatted at 2026-09-29 19:06:42 on dist-test-slave-nmsz"
server_key: "bbb2b07ce197963cd02d676ab03262f2"
server_key_iv: "e2155c868575955bd8a69302df37cb43"
server_key_version: "encryptionkey@0"
I20260929 19:06:42.786113 25778 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 19:06:42.786973 25793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:42.787168 25778 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:06:42.787263 25778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "42e37ba5a9134c168ae58e2676188068"
format_stamp: "Formatted at 2026-09-29 19:06:42 on dist-test-slave-nmsz"
server_key: "bbb2b07ce197963cd02d676ab03262f2"
server_key_iv: "e2155c868575955bd8a69302df37cb43"
server_key_version: "encryptionkey@0"
I20260929 19:06:42.787349 25778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:06:42.808602 25778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:06:42.810519 25778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:06:42.810678 25778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:06:42.814601 25778 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.126:42189
I20260929 19:06:42.814649 25845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.126:42189 every 8 connection(s)
I20260929 19:06:42.814996 25778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:06:42.815836 25846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:06:42.817127 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 25778
I20260929 19:06:42.817225 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:06:42.817418 8825 external_mini_cluster.cc:1428] Setting key 91989a56cbbdbc16fa074d409a1848d8
I20260929 19:06:42.818449 25846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42e37ba5a9134c168ae58e2676188068: Bootstrap starting.
I20260929 19:06:42.818987 25846 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42e37ba5a9134c168ae58e2676188068: Neither blocks nor log segments found. Creating new log.
I20260929 19:06:42.819267 25846 log.cc:824] T 00000000000000000000000000000000 P 42e37ba5a9134c168ae58e2676188068: Log is configured to *not* fsync() on all Append() calls
I20260929 19:06:42.820029 25846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42e37ba5a9134c168ae58e2676188068: No bootstrap required, opened a new log
I20260929 19:06:42.822185 25846 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42e37ba5a9134c168ae58e2676188068 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42e37ba5a9134c168ae58e2676188068" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 42189 } }
I20260929 19:06:42.822371 25846 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42e37ba5a9134c168ae58e2676188068 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:06:42.822443 25846 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42e37ba5a9134c168ae58e2676188068 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42e37ba5a9134c168ae58e2676188068, State: Initialized, Role: FOLLOWER
I20260929 19:06:42.822597 25846 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42e37ba5a9134c168ae58e2676188068 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42e37ba5a9134c168ae58e2676188068" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 42189 } }
I20260929 19:06:42.822692 25846 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42e37ba5a9134c168ae58e2676188068 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:06:42.822726 25846 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42e37ba5a9134c168ae58e2676188068 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:06:42.822772 25846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42e37ba5a9134c168ae58e2676188068 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:06:42.823298 25846 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42e37ba5a9134c168ae58e2676188068 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42e37ba5a9134c168ae58e2676188068" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 42189 } }
I20260929 19:06:42.823385 25846 leader_election.cc:302] T 00000000000000000000000000000000 P 42e37ba5a9134c168ae58e2676188068 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42e37ba5a9134c168ae58e2676188068; no voters:
I20260929 19:06:42.823557 25846 leader_election.cc:288] T 00000000000000000000000000000000 P 42e37ba5a9134c168ae58e2676188068 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:06:42.823616 25851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42e37ba5a9134c168ae58e2676188068 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:06:42.823764 25851 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42e37ba5a9134c168ae58e2676188068 [term 1 LEADER]: Becoming Leader. State: Replica: 42e37ba5a9134c168ae58e2676188068, State: Running, Role: LEADER
I20260929 19:06:42.823837 25846 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42e37ba5a9134c168ae58e2676188068 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:06:42.823920 25851 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42e37ba5a9134c168ae58e2676188068 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42e37ba5a9134c168ae58e2676188068" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 42189 } }
I20260929 19:06:42.824309 25851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42e37ba5a9134c168ae58e2676188068 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42e37ba5a9134c168ae58e2676188068. Latest consensus state: current_term: 1 leader_uuid: "42e37ba5a9134c168ae58e2676188068" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42e37ba5a9134c168ae58e2676188068" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 42189 } } }
I20260929 19:06:42.824451 25851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42e37ba5a9134c168ae58e2676188068 [sys.catalog]: This master's current role is: LEADER
Sep 29 19:06:42 dist-test-slave-nmsz krb5kdc[25765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:06:42.824985 25860 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:06:42.825016 25853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42e37ba5a9134c168ae58e2676188068 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42e37ba5a9134c168ae58e2676188068" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42e37ba5a9134c168ae58e2676188068" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 42189 } } }
I20260929 19:06:42.825598 25853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42e37ba5a9134c168ae58e2676188068 [sys.catalog]: This master's current role is: LEADER
I20260929 19:06:42.825743 25860 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:06:42.827316 25860 catalog_manager.cc:1347] Generated new cluster ID: 3c034961f8bb46c5899a687f11d9bbd2
I20260929 19:06:42.827369 25860 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:06:42.828671 25849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:42.817923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52662 (local address 127.8.158.126:42189)
0929 19:06:42.818123 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:42.818132 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:42.818165 (+ 33us) server_negotiation.cc:407] Connection header received
0929 19:06:42.818677 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:42.818684 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:42.818741 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:42.818888 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:42.819608 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:42.820949 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:42.821662 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:42.822301 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:42.822423 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:42.825230 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:42.825233 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:42.825234 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:06:42.825245 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:42.827082 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:42.827471 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:42.827473 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:42.827474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:42.827574 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:42.827802 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:42.827803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:42.827803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:42.828091 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:42.828229 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:42.828329 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 19:06:42.828475 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":99,"threads_started":1}
I20260929 19:06:42.833746 25860 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:06:42.834237 25860 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:06:42.838824 25860 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42e37ba5a9134c168ae58e2676188068: Generated new TSK 0
I20260929 19:06:42.838987 25860 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.158.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab.
Entry for principal kudu/127.8.158.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab.
I20260929 19:06:42.865177 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.65:0
--local_ip_for_outbound_sockets=127.8.158.65
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:42189
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab
--principal=kudu/127.8.158.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:41705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:06:42.944787 25872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:06:42.945032 25872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:06:42.945068 25872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:06:42.946213 25872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:06:42.946282 25872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:06:42.946324 25872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.65
I20260929 19:06:42.947805 25872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:41705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab
--principal=kudu/127.8.158.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:42189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.65
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:06:42.948071 25872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:06:42.948307 25872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:06:42.953118 25880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:42.953166 25877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:42.953166 25878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:42.953366 25872 server_base.cc:1034] running on GCE node
I20260929 19:06:42.953594 25872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:06:42.953823 25872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:06:42.954962 25872 hybrid_clock.cc:648] HybridClock initialized: now 1790708802954944 us; error 37 us; skew 500 ppm
Sep 29 19:06:42 dist-test-slave-nmsz krb5kdc[25765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708802, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:06:42.957389 25872 init.cc:377] Logged in from keytab as kudu/127.8.158.65@KRBTEST.COM (short username kudu)
I20260929 19:06:42.957795 25872 webserver.cc:460] Webserver started at http://127.8.158.65:32879/ using document root <none> and password file <none>
I20260929 19:06:42.958004 25872 fs_manager.cc:359] Metadata directory not provided
I20260929 19:06:42.958073 25872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:06:42.958220 25872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:06:42.959239 25872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-0/data/instance:
uuid: "eff2e38242d94402a1e5479ad01c8eb3"
format_stamp: "Formatted at 2026-09-29 19:06:42 on dist-test-slave-nmsz"
server_key: "cec182a567170506651f886d0bbedb71"
server_key_iv: "4e1a5a4734605942b8c4e106637e47a3"
server_key_version: "encryptionkey@0"
I20260929 19:06:42.959606 25872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance:
uuid: "eff2e38242d94402a1e5479ad01c8eb3"
format_stamp: "Formatted at 2026-09-29 19:06:42 on dist-test-slave-nmsz"
server_key: "cec182a567170506651f886d0bbedb71"
server_key_iv: "4e1a5a4734605942b8c4e106637e47a3"
server_key_version: "encryptionkey@0"
I20260929 19:06:42.960852 25872 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 19:06:42.961542 25887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:42.961710 25872 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:06:42.961802 25872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-0/wal
uuid: "eff2e38242d94402a1e5479ad01c8eb3"
format_stamp: "Formatted at 2026-09-29 19:06:42 on dist-test-slave-nmsz"
server_key: "cec182a567170506651f886d0bbedb71"
server_key_iv: "4e1a5a4734605942b8c4e106637e47a3"
server_key_version: "encryptionkey@0"
I20260929 19:06:42.961879 25872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:06:42.990005 25872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:06:42.992363 25872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:06:42.992472 25872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:06:42.992646 25872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:06:42.992949 25872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:06:42.993005 25872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:42.993046 25872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:06:42.993070 25872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:42.998643 25872 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.65:42561
I20260929 19:06:42.998744 26000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.65:42561 every 8 connection(s)
I20260929 19:06:42.999022 25872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
I20260929 19:06:43.000069 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 25872
I20260929 19:06:43.000149 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance
I20260929 19:06:43.000355 8825 external_mini_cluster.cc:1428] Setting key e4eba88f4d3d2f2c4f35a2472194f15b
Sep 29 19:06:43 dist-test-slave-nmsz krb5kdc[25765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708802, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.158.66@KRBTEST.COM; defaulting to no policy
I20260929 19:06:43.009248 25849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:42.999765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:45849 (local address 127.8.158.126:42189)
0929 19:06:42.999831 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:42.999833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:42.999958 (+ 125us) server_negotiation.cc:407] Connection header received
0929 19:06:43.000523 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:43.000524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:43.000536 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:43.000637 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:43.001602 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.002419 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.003895 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.004088 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.004139 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:43.006499 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:43.006501 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:43.006501 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:06:43.006509 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:43.007870 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:43.008278 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:43.008279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:43.008280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:43.008339 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:43.008592 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:43.008593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:43.008594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:43.008713 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:43.008796 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:43.009062 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 19:06:43.009158 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 19:06:43.009399 26003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:42.999672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:42189 (local address 127.8.158.65:45849)
0929 19:06:42.999826 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:42.999864 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:43.000429 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:43.000690 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:43.000699 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:43.001308 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:06:43.001492 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:43.001497 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.002514 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.002515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:43.003806 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:43.003809 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.004169 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.004170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:43.004315 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:43.004783 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:06:43.004793 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:06:43.006408 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:06:43.007973 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:06:43.007979 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:06:43.007980 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:06:43.008190 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:06:43.008421 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:06:43.008421 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:06:43.008422 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:06:43.008526 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:06:43.008796 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:06:43.008804 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:06:43.008935 (+ 131us) client_negotiation.cc:769] Sending connection context
0929 19:06:43.009041 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 19:06:43.009182 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":89,"threads_started":1}
I20260929 19:06:43.009898 26001 heartbeater.cc:347] Connected to a master server at 127.8.158.126:42189
I20260929 19:06:43.010015 26001 heartbeater.cc:464] Registering TS with master...
I20260929 19:06:43.010222 26001 heartbeater.cc:511] Master 127.8.158.126:42189 requested a full tablet report, sending...
I20260929 19:06:43.010802 25810 ts_manager.cc:194] Registered new tserver with Master: eff2e38242d94402a1e5479ad01c8eb3 (127.8.158.65:42561)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.66@KRBTEST.COM" created.
I20260929 19:06:43.011453 25810 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.65@KRBTEST.COM'} at 127.8.158.65:45849
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab.
Entry for principal kudu/127.8.158.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab.
I20260929 19:06:43.025564 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.66:0
--local_ip_for_outbound_sockets=127.8.158.66
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:42189
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab
--principal=kudu/127.8.158.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:41705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:06:43.104300 26006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:06:43.104487 26006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:06:43.104511 26006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:06:43.105695 26006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:06:43.105736 26006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:06:43.105789 26006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.66
I20260929 19:06:43.107301 26006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:41705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab
--principal=kudu/127.8.158.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:42189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.66
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:06:43.107532 26006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:06:43.107751 26006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:06:43.112501 26012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:43.112603 26006 server_base.cc:1034] running on GCE node
W20260929 19:06:43.112516 26011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:43.112560 26014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:43.112996 26006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:06:43.113188 26006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:06:43.114378 26006 hybrid_clock.cc:648] HybridClock initialized: now 1790708803114344 us; error 40 us; skew 500 ppm
Sep 29 19:06:43 dist-test-slave-nmsz krb5kdc[25765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708803, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:06:43.116832 26006 init.cc:377] Logged in from keytab as kudu/127.8.158.66@KRBTEST.COM (short username kudu)
I20260929 19:06:43.117264 26006 webserver.cc:460] Webserver started at http://127.8.158.66:42979/ using document root <none> and password file <none>
I20260929 19:06:43.117481 26006 fs_manager.cc:359] Metadata directory not provided
I20260929 19:06:43.117527 26006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:06:43.117663 26006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:06:43.118728 26006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-1/data/instance:
uuid: "48b80b6f1c5446d79d93247980ac1f78"
format_stamp: "Formatted at 2026-09-29 19:06:43 on dist-test-slave-nmsz"
server_key: "7986c0e9f4395a06c6dcdef99b74c8dc"
server_key_iv: "57d342143bec3683c7bd5a1b8a386074"
server_key_version: "encryptionkey@0"
I20260929 19:06:43.119081 26006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance:
uuid: "48b80b6f1c5446d79d93247980ac1f78"
format_stamp: "Formatted at 2026-09-29 19:06:43 on dist-test-slave-nmsz"
server_key: "7986c0e9f4395a06c6dcdef99b74c8dc"
server_key_iv: "57d342143bec3683c7bd5a1b8a386074"
server_key_version: "encryptionkey@0"
I20260929 19:06:43.120340 26006 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:06:43.121027 26021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:43.121201 26006 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:06:43.121295 26006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-1/wal
uuid: "48b80b6f1c5446d79d93247980ac1f78"
format_stamp: "Formatted at 2026-09-29 19:06:43 on dist-test-slave-nmsz"
server_key: "7986c0e9f4395a06c6dcdef99b74c8dc"
server_key_iv: "57d342143bec3683c7bd5a1b8a386074"
server_key_version: "encryptionkey@0"
I20260929 19:06:43.121378 26006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:06:43.137382 26006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:06:43.139515 26006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:06:43.139673 26006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:06:43.139885 26006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:06:43.140178 26006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:06:43.140231 26006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:43.140265 26006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:06:43.140291 26006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:43.145857 26006 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.66:41907
I20260929 19:06:43.145960 26134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.66:41907 every 8 connection(s)
I20260929 19:06:43.146226 26006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
I20260929 19:06:43.149514 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 26006
I20260929 19:06:43.149647 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance
I20260929 19:06:43.149820 8825 external_mini_cluster.cc:1428] Setting key 53aceac3de13702cecf6f4d3b15ee2f6
Sep 29 19:06:43 dist-test-slave-nmsz krb5kdc[25765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708803, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.66@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:06:43.155977 26137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:43.146886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:42189 (local address 127.8.158.66:49585)
0929 19:06:43.147043 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:43.147082 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:43.147522 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:43.147710 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:43.147713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:43.148268 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:06:43.148442 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:43.148446 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.149102 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.149103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:43.150075 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:43.150078 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.150599 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.150600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:43.150740 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:43.151231 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:06:43.151239 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:06:43.152874 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:06:43.154732 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:06:43.154737 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:06:43.154739 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:06:43.154940 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:06:43.155167 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:06:43.155167 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:06:43.155167 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:06:43.155257 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:06:43.155547 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:06:43.155555 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:06:43.155651 (+ 96us) client_negotiation.cc:769] Sending connection context
0929 19:06:43.155726 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 19:06:43.155831 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":93,"threads_started":1}
I20260929 19:06:43.155970 25849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:43.146911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:49585 (local address 127.8.158.126:42189)
0929 19:06:43.146972 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:43.146974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:43.147166 (+ 192us) server_negotiation.cc:407] Connection header received
0929 19:06:43.147598 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:43.147599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:43.147608 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:43.147663 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:43.148527 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.149002 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.150182 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.150513 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.150590 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:43.152974 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:43.152975 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:43.152976 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:06:43.152982 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:43.154652 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:43.155017 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:43.155018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:43.155019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:43.155095 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:43.155323 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:43.155324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:43.155324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:43.155477 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:43.155555 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:43.155776 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 19:06:43.155868 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 19:06:43.156428 26135 heartbeater.cc:347] Connected to a master server at 127.8.158.126:42189
I20260929 19:06:43.156526 26135 heartbeater.cc:464] Registering TS with master...
I20260929 19:06:43.156709 26135 heartbeater.cc:511] Master 127.8.158.126:42189 requested a full tablet report, sending...
I20260929 19:06:43.157100 25810 ts_manager.cc:194] Registered new tserver with Master: 48b80b6f1c5446d79d93247980ac1f78 (127.8.158.66:41907)
I20260929 19:06:43.157632 25810 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.66@KRBTEST.COM'} at 127.8.158.66:49585
WARNING: no policy specified for kudu/127.8.158.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab.
Entry for principal kudu/127.8.158.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab.
I20260929 19:06:43.174911 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.67:0
--local_ip_for_outbound_sockets=127.8.158.67
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:42189
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab
--principal=kudu/127.8.158.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:41705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:06:43.252413 26140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:06:43.252596 26140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:06:43.252619 26140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:06:43.253722 26140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:06:43.253762 26140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:06:43.253787 26140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.67
I20260929 19:06:43.255195 26140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:41705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab
--principal=kudu/127.8.158.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:42189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.67
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:06:43.255419 26140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:06:43.255643 26140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:06:43.260365 26146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:43.260497 26145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:43.260623 26148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:43.260681 26140 server_base.cc:1034] running on GCE node
I20260929 19:06:43.261010 26140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:06:43.261250 26140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:06:43.262372 26140 hybrid_clock.cc:648] HybridClock initialized: now 1790708803262363 us; error 36 us; skew 500 ppm
Sep 29 19:06:43 dist-test-slave-nmsz krb5kdc[25765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708803, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:06:43.265626 26140 init.cc:377] Logged in from keytab as kudu/127.8.158.67@KRBTEST.COM (short username kudu)
I20260929 19:06:43.266148 26140 webserver.cc:460] Webserver started at http://127.8.158.67:43203/ using document root <none> and password file <none>
I20260929 19:06:43.266459 26140 fs_manager.cc:359] Metadata directory not provided
I20260929 19:06:43.266525 26140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:06:43.266657 26140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:06:43.267791 26140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-2/data/instance:
uuid: "cf2c2991983b405f9a967e56d29564e7"
format_stamp: "Formatted at 2026-09-29 19:06:43 on dist-test-slave-nmsz"
server_key: "41618c50a09a76646ac10ab907783cd3"
server_key_iv: "7b197d062a6988e329a83d205e7a6e19"
server_key_version: "encryptionkey@0"
I20260929 19:06:43.268086 26140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance:
uuid: "cf2c2991983b405f9a967e56d29564e7"
format_stamp: "Formatted at 2026-09-29 19:06:43 on dist-test-slave-nmsz"
server_key: "41618c50a09a76646ac10ab907783cd3"
server_key_iv: "7b197d062a6988e329a83d205e7a6e19"
server_key_version: "encryptionkey@0"
I20260929 19:06:43.269282 26140 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:06:43.270139 26155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:43.270357 26140 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:06:43.270474 26140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-2/wal
uuid: "cf2c2991983b405f9a967e56d29564e7"
format_stamp: "Formatted at 2026-09-29 19:06:43 on dist-test-slave-nmsz"
server_key: "41618c50a09a76646ac10ab907783cd3"
server_key_iv: "7b197d062a6988e329a83d205e7a6e19"
server_key_version: "encryptionkey@0"
I20260929 19:06:43.270550 26140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:06:43.286005 26140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:06:43.288221 26140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:06:43.288385 26140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:06:43.288599 26140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:06:43.288888 26140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:06:43.288941 26140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:43.288975 26140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:06:43.289005 26140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:43.295109 26140 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.67:45317
I20260929 19:06:43.295157 26268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.67:45317 every 8 connection(s)
I20260929 19:06:43.295441 26140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
I20260929 19:06:43.298981 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 26140
I20260929 19:06:43.299077 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance
I20260929 19:06:43.299304 8825 external_mini_cluster.cc:1428] Setting key 6b4ba67a8ab05c4e40eb20932d5216f9
Sep 29 19:06:43 dist-test-slave-nmsz krb5kdc[25765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708803, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.67@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:06:43.304617 25849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:43.296136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:44179 (local address 127.8.158.126:42189)
0929 19:06:43.296195 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:43.296196 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:43.296384 (+ 188us) server_negotiation.cc:407] Connection header received
0929 19:06:43.296836 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:43.296837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:43.296848 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:43.296945 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:43.297793 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.298452 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.299587 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.299923 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.299987 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:43.302033 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:43.302034 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:43.302034 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:06:43.302041 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:43.303378 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:43.303776 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:43.303777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:43.303777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:43.303828 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:43.304076 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:43.304076 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:43.304076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:43.304181 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:43.304232 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:43.304499 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 19:06:43.304558 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 19:06:43.304817 26271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:43.296131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:42189 (local address 127.8.158.67:44179)
0929 19:06:43.296287 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:43.296319 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:43.296752 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:43.297007 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:43.297010 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:43.297532 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:06:43.297705 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:43.297708 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.298535 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.298536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:43.299500 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:43.299503 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.300010 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.300011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:43.300166 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:43.300557 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:06:43.300564 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:06:43.301877 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:06:43.303467 (+ 1590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:06:43.303473 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:06:43.303474 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:06:43.303702 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:06:43.303918 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:06:43.303918 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:06:43.303919 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:06:43.304017 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:06:43.304264 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:06:43.304269 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:06:43.304382 (+ 113us) client_negotiation.cc:769] Sending connection context
0929 19:06:43.304471 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 19:06:43.304580 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":96,"threads_started":1}
I20260929 19:06:43.305322 26269 heartbeater.cc:347] Connected to a master server at 127.8.158.126:42189
I20260929 19:06:43.305400 26269 heartbeater.cc:464] Registering TS with master...
I20260929 19:06:43.305522 26269 heartbeater.cc:511] Master 127.8.158.126:42189 requested a full tablet report, sending...
I20260929 19:06:43.305831 25810 ts_manager.cc:194] Registered new tserver with Master: cf2c2991983b405f9a967e56d29564e7 (127.8.158.67:45317)
I20260929 19:06:43.306159 25810 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.67@KRBTEST.COM'} at 127.8.158.67:44179
I20260929 19:06:43.311061 8825 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:06:43.318027 25849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:43.312377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52666 (local address 127.8.158.126:42189)
0929 19:06:43.312452 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:43.312455 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:43.312552 (+ 97us) server_negotiation.cc:407] Connection header received
0929 19:06:43.312638 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:43.312639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:43.312647 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:43.312695 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:43.313227 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.313683 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.314425 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.314624 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.314705 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:43.315790 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:43.315792 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:43.315792 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:06:43.315798 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:43.316905 (+ 1107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:43.317244 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:43.317245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:43.317245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:43.317296 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:43.317507 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:43.317508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:43.317509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:43.317644 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:43.317749 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:43.317848 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 19:06:43.317932 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 19:06:43.319283 25810 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52666:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:06:43.320698 25810 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:06:43.327013 26280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:43.322779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35544 (local address 127.8.158.66:41907)
0929 19:06:43.322967 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:43.322971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:43.323338 (+ 367us) server_negotiation.cc:407] Connection header received
0929 19:06:43.323378 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:43.323379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:43.323393 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:43.323480 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:06:43.323720 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.324607 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.326022 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.326602 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.326689 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:43.326745 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:43.326895 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 19:06:43.326940 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":70,"threads_started":1}
I20260929 19:06:43.327005 26277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:43.322645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.66:41907 (local address 127.0.0.1:35544)
0929 19:06:43.323228 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:43.323237 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:43.323316 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:43.323592 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:43.323594 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:43.323596 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:06:43.323645 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:43.323647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.324713 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.324714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:43.325935 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:43.325938 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.326689 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.326690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:43.326768 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:43.326780 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 19:06:43.326880 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 19:06:43.326931 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":52,"threads_started":1}
I20260929 19:06:43.327208 26278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:43.322640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.67:45317 (local address 127.0.0.1:33750)
0929 19:06:43.322791 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:43.322828 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:43.322888 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:43.323167 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:43.323169 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:43.323170 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:06:43.323211 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:43.323212 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.324554 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.324555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:43.326010 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:43.326012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.326969 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.326970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:43.327033 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:43.327038 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 19:06:43.327088 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 19:06:43.327130 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":9,"thread_start_us":62,"threads_started":1}
I20260929 19:06:43.327292 26282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:43.322878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51940 (local address 127.8.158.65:42561)
0929 19:06:43.323071 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:43.323074 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:43.323465 (+ 391us) server_negotiation.cc:407] Connection header received
0929 19:06:43.323554 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:43.323557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:43.323573 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:43.323647 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:06:43.323825 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.324824 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.326072 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.326811 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.326879 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:43.326937 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:43.327169 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 19:06:43.327206 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":84,"threads_started":1}
I20260929 19:06:43.327478 26281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:43.322794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33750 (local address 127.8.158.67:45317)
0929 19:06:43.323011 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:43.323016 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:43.323029 (+ 13us) server_negotiation.cc:407] Connection header received
0929 19:06:43.323066 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:43.323069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:43.323084 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:43.323166 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:06:43.323287 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.324426 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.326092 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.326812 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.326879 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:43.326931 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:43.327361 (+ 430us) server_negotiation.cc:299] Negotiation successful
0929 19:06:43.327412 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":72,"threads_started":1}
I20260929 19:06:43.327631 26279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:43.322653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:42561 (local address 127.0.0.1:51940)
0929 19:06:43.323386 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:43.323395 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:43.323441 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:43.323727 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:43.323728 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:43.323730 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:06:43.323762 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:43.323763 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.324931 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.324932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:43.325965 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:43.325967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.326968 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.326969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:43.327033 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:43.327037 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:06:43.327088 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 19:06:43.327132 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"mutex_wait_us":4,"thread_start_us":49,"threads_started":1}
I20260929 19:06:43.328088 26203 tablet_service.cc:1467] Processing CreateTablet for tablet 0300041805224e22a8758ea1ee3a8646 (DEFAULT_TABLE table=test-table [id=fe609663bb6e47caa2b32a6c4862a8f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:06:43.328118 26069 tablet_service.cc:1467] Processing CreateTablet for tablet 0300041805224e22a8758ea1ee3a8646 (DEFAULT_TABLE table=test-table [id=fe609663bb6e47caa2b32a6c4862a8f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:06:43.328243 25935 tablet_service.cc:1467] Processing CreateTablet for tablet 0300041805224e22a8758ea1ee3a8646 (DEFAULT_TABLE table=test-table [id=fe609663bb6e47caa2b32a6c4862a8f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:06:43.328423 26203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0300041805224e22a8758ea1ee3a8646. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:06:43.328451 26069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0300041805224e22a8758ea1ee3a8646. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:06:43.328513 25935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0300041805224e22a8758ea1ee3a8646. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:06:43.330819 26283 tablet_bootstrap.cc:492] T 0300041805224e22a8758ea1ee3a8646 P cf2c2991983b405f9a967e56d29564e7: Bootstrap starting.
I20260929 19:06:43.331391 26283 tablet_bootstrap.cc:654] T 0300041805224e22a8758ea1ee3a8646 P cf2c2991983b405f9a967e56d29564e7: Neither blocks nor log segments found. Creating new log.
I20260929 19:06:43.331615 26284 tablet_bootstrap.cc:492] T 0300041805224e22a8758ea1ee3a8646 P eff2e38242d94402a1e5479ad01c8eb3: Bootstrap starting.
I20260929 19:06:43.332005 26283 log.cc:824] T 0300041805224e22a8758ea1ee3a8646 P cf2c2991983b405f9a967e56d29564e7: Log is configured to *not* fsync() on all Append() calls
I20260929 19:06:43.332158 26285 tablet_bootstrap.cc:492] T 0300041805224e22a8758ea1ee3a8646 P 48b80b6f1c5446d79d93247980ac1f78: Bootstrap starting.
I20260929 19:06:43.332293 26284 tablet_bootstrap.cc:654] T 0300041805224e22a8758ea1ee3a8646 P eff2e38242d94402a1e5479ad01c8eb3: Neither blocks nor log segments found. Creating new log.
I20260929 19:06:43.332738 26283 tablet_bootstrap.cc:492] T 0300041805224e22a8758ea1ee3a8646 P cf2c2991983b405f9a967e56d29564e7: No bootstrap required, opened a new log
I20260929 19:06:43.332768 26284 log.cc:824] T 0300041805224e22a8758ea1ee3a8646 P eff2e38242d94402a1e5479ad01c8eb3: Log is configured to *not* fsync() on all Append() calls
I20260929 19:06:43.332824 26285 tablet_bootstrap.cc:654] T 0300041805224e22a8758ea1ee3a8646 P 48b80b6f1c5446d79d93247980ac1f78: Neither blocks nor log segments found. Creating new log.
I20260929 19:06:43.332820 26283 ts_tablet_manager.cc:1397] T 0300041805224e22a8758ea1ee3a8646 P cf2c2991983b405f9a967e56d29564e7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:06:43.333369 26285 log.cc:824] T 0300041805224e22a8758ea1ee3a8646 P 48b80b6f1c5446d79d93247980ac1f78: Log is configured to *not* fsync() on all Append() calls
I20260929 19:06:43.333482 26284 tablet_bootstrap.cc:492] T 0300041805224e22a8758ea1ee3a8646 P eff2e38242d94402a1e5479ad01c8eb3: No bootstrap required, opened a new log
I20260929 19:06:43.333549 26284 ts_tablet_manager.cc:1397] T 0300041805224e22a8758ea1ee3a8646 P eff2e38242d94402a1e5479ad01c8eb3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:06:43.334059 26285 tablet_bootstrap.cc:492] T 0300041805224e22a8758ea1ee3a8646 P 48b80b6f1c5446d79d93247980ac1f78: No bootstrap required, opened a new log
I20260929 19:06:43.334127 26285 ts_tablet_manager.cc:1397] T 0300041805224e22a8758ea1ee3a8646 P 48b80b6f1c5446d79d93247980ac1f78: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:06:43.335116 26283 raft_consensus.cc:348] T 0300041805224e22a8758ea1ee3a8646 P cf2c2991983b405f9a967e56d29564e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b80b6f1c5446d79d93247980ac1f78" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 41907 } } peers { permanent_uuid: "cf2c2991983b405f9a967e56d29564e7" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45317 } } peers { permanent_uuid: "eff2e38242d94402a1e5479ad01c8eb3" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 42561 } }
I20260929 19:06:43.335167 26284 raft_consensus.cc:348] T 0300041805224e22a8758ea1ee3a8646 P eff2e38242d94402a1e5479ad01c8eb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b80b6f1c5446d79d93247980ac1f78" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 41907 } } peers { permanent_uuid: "cf2c2991983b405f9a967e56d29564e7" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45317 } } peers { permanent_uuid: "eff2e38242d94402a1e5479ad01c8eb3" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 42561 } }
I20260929 19:06:43.335289 26283 raft_consensus.cc:374] T 0300041805224e22a8758ea1ee3a8646 P cf2c2991983b405f9a967e56d29564e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:06:43.335309 26284 raft_consensus.cc:374] T 0300041805224e22a8758ea1ee3a8646 P eff2e38242d94402a1e5479ad01c8eb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:06:43.335326 26283 raft_consensus.cc:729] T 0300041805224e22a8758ea1ee3a8646 P cf2c2991983b405f9a967e56d29564e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf2c2991983b405f9a967e56d29564e7, State: Initialized, Role: FOLLOWER
I20260929 19:06:43.335374 26284 raft_consensus.cc:729] T 0300041805224e22a8758ea1ee3a8646 P eff2e38242d94402a1e5479ad01c8eb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff2e38242d94402a1e5479ad01c8eb3, State: Initialized, Role: FOLLOWER
I20260929 19:06:43.335454 26283 consensus_queue.cc:260] T 0300041805224e22a8758ea1ee3a8646 P cf2c2991983b405f9a967e56d29564e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b80b6f1c5446d79d93247980ac1f78" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 41907 } } peers { permanent_uuid: "cf2c2991983b405f9a967e56d29564e7" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45317 } } peers { permanent_uuid: "eff2e38242d94402a1e5479ad01c8eb3" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 42561 } }
I20260929 19:06:43.335502 26284 consensus_queue.cc:260] T 0300041805224e22a8758ea1ee3a8646 P eff2e38242d94402a1e5479ad01c8eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b80b6f1c5446d79d93247980ac1f78" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 41907 } } peers { permanent_uuid: "cf2c2991983b405f9a967e56d29564e7" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45317 } } peers { permanent_uuid: "eff2e38242d94402a1e5479ad01c8eb3" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 42561 } }
I20260929 19:06:43.335732 26283 ts_tablet_manager.cc:1428] T 0300041805224e22a8758ea1ee3a8646 P cf2c2991983b405f9a967e56d29564e7: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260929 19:06:43.335793 26284 ts_tablet_manager.cc:1428] T 0300041805224e22a8758ea1ee3a8646 P eff2e38242d94402a1e5479ad01c8eb3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:06:43.335736 26269 heartbeater.cc:503] Master 127.8.158.126:42189 was elected leader, sending a full tablet report...
I20260929 19:06:43.336071 26001 heartbeater.cc:503] Master 127.8.158.126:42189 was elected leader, sending a full tablet report...
I20260929 19:06:43.336108 26285 raft_consensus.cc:348] T 0300041805224e22a8758ea1ee3a8646 P 48b80b6f1c5446d79d93247980ac1f78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b80b6f1c5446d79d93247980ac1f78" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 41907 } } peers { permanent_uuid: "cf2c2991983b405f9a967e56d29564e7" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45317 } } peers { permanent_uuid: "eff2e38242d94402a1e5479ad01c8eb3" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 42561 } }
I20260929 19:06:43.336254 26285 raft_consensus.cc:374] T 0300041805224e22a8758ea1ee3a8646 P 48b80b6f1c5446d79d93247980ac1f78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:06:43.336290 26285 raft_consensus.cc:729] T 0300041805224e22a8758ea1ee3a8646 P 48b80b6f1c5446d79d93247980ac1f78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48b80b6f1c5446d79d93247980ac1f78, State: Initialized, Role: FOLLOWER
I20260929 19:06:43.336391 26285 consensus_queue.cc:260] T 0300041805224e22a8758ea1ee3a8646 P 48b80b6f1c5446d79d93247980ac1f78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b80b6f1c5446d79d93247980ac1f78" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 41907 } } peers { permanent_uuid: "cf2c2991983b405f9a967e56d29564e7" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45317 } } peers { permanent_uuid: "eff2e38242d94402a1e5479ad01c8eb3" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 42561 } }
I20260929 19:06:43.336663 26285 ts_tablet_manager.cc:1428] T 0300041805224e22a8758ea1ee3a8646 P 48b80b6f1c5446d79d93247980ac1f78: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:06:43.336781 26135 heartbeater.cc:503] Master 127.8.158.126:42189 was elected leader, sending a full tablet report...
I20260929 19:06:43.349864 26290 raft_consensus.cc:482] T 0300041805224e22a8758ea1ee3a8646 P eff2e38242d94402a1e5479ad01c8eb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:06:43.349961 26290 raft_consensus.cc:504] T 0300041805224e22a8758ea1ee3a8646 P eff2e38242d94402a1e5479ad01c8eb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b80b6f1c5446d79d93247980ac1f78" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 41907 } } peers { permanent_uuid: "cf2c2991983b405f9a967e56d29564e7" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45317 } } peers { permanent_uuid: "eff2e38242d94402a1e5479ad01c8eb3" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 42561 } }
I20260929 19:06:43.350298 26290 leader_election.cc:288] T 0300041805224e22a8758ea1ee3a8646 P eff2e38242d94402a1e5479ad01c8eb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48b80b6f1c5446d79d93247980ac1f78 (127.8.158.66:41907), cf2c2991983b405f9a967e56d29564e7 (127.8.158.67:45317)
I20260929 19:06:43.353466 26280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:43.350637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:57095 (local address 127.8.158.66:41907)
0929 19:06:43.350681 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:43.350683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:43.350690 (+ 7us) server_negotiation.cc:407] Connection header received
0929 19:06:43.350713 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:43.350714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:43.350723 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:43.350825 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:06:43.351072 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.351601 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.352718 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.353108 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.353163 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:43.353169 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:43.353356 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 19:06:43.353397 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 19:06:43.353497 26281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:43.350597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:34155 (local address 127.8.158.67:45317)
0929 19:06:43.350677 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:43.350679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:43.350963 (+ 284us) server_negotiation.cc:407] Connection header received
0929 19:06:43.350991 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:43.350992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:43.351002 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:43.351058 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:06:43.351235 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.351768 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.352713 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.353116 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.353214 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:43.353222 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:43.353395 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 19:06:43.353434 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 19:06:43.353497 26003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:43.350512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.66:41907 (local address 127.8.158.65:57095)
0929 19:06:43.350587 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:43.350600 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:43.350650 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:43.350868 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:43.350869 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:43.350872 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:06:43.350986 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:43.350988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.351679 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.351680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:43.352618 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:43.352621 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.353189 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.353190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:43.353283 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:43.353286 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:06:43.353335 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 19:06:43.353403 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":1}
I20260929 19:06:43.353515 26292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:43.350512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.67:45317 (local address 127.8.158.65:34155)
0929 19:06:43.350850 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:43.350860 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:43.350909 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:43.351072 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:43.351073 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:43.351074 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:06:43.351156 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:43.351158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.351853 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.351854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:43.352618 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:43.352620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.353225 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.353225 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:43.353299 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:43.353302 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:06:43.353375 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 19:06:43.353418 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":160,"thread_start_us":91,"threads_started":1}
I20260929 19:06:43.353889 26089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0300041805224e22a8758ea1ee3a8646" candidate_uuid: "eff2e38242d94402a1e5479ad01c8eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48b80b6f1c5446d79d93247980ac1f78" is_pre_election: true
I20260929 19:06:43.353968 26223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0300041805224e22a8758ea1ee3a8646" candidate_uuid: "eff2e38242d94402a1e5479ad01c8eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf2c2991983b405f9a967e56d29564e7" is_pre_election: true
I20260929 19:06:43.354035 26089 raft_consensus.cc:2454] T 0300041805224e22a8758ea1ee3a8646 P 48b80b6f1c5446d79d93247980ac1f78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff2e38242d94402a1e5479ad01c8eb3 in term 0.
I20260929 19:06:43.354116 26223 raft_consensus.cc:2454] T 0300041805224e22a8758ea1ee3a8646 P cf2c2991983b405f9a967e56d29564e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff2e38242d94402a1e5479ad01c8eb3 in term 0.
I20260929 19:06:43.354256 25889 leader_election.cc:302] T 0300041805224e22a8758ea1ee3a8646 P eff2e38242d94402a1e5479ad01c8eb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48b80b6f1c5446d79d93247980ac1f78, eff2e38242d94402a1e5479ad01c8eb3; no voters:
I20260929 19:06:43.354426 26290 raft_consensus.cc:2781] T 0300041805224e22a8758ea1ee3a8646 P eff2e38242d94402a1e5479ad01c8eb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:06:43.354487 26290 raft_consensus.cc:482] T 0300041805224e22a8758ea1ee3a8646 P eff2e38242d94402a1e5479ad01c8eb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:06:43.354526 26290 raft_consensus.cc:3037] T 0300041805224e22a8758ea1ee3a8646 P eff2e38242d94402a1e5479ad01c8eb3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:06:43.355206 26290 raft_consensus.cc:504] T 0300041805224e22a8758ea1ee3a8646 P eff2e38242d94402a1e5479ad01c8eb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b80b6f1c5446d79d93247980ac1f78" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 41907 } } peers { permanent_uuid: "cf2c2991983b405f9a967e56d29564e7" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45317 } } peers { permanent_uuid: "eff2e38242d94402a1e5479ad01c8eb3" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 42561 } }
I20260929 19:06:43.355345 26290 leader_election.cc:288] T 0300041805224e22a8758ea1ee3a8646 P eff2e38242d94402a1e5479ad01c8eb3 [CANDIDATE]: Term 1 election: Requested vote from peers 48b80b6f1c5446d79d93247980ac1f78 (127.8.158.66:41907), cf2c2991983b405f9a967e56d29564e7 (127.8.158.67:45317)
I20260929 19:06:43.355526 26223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0300041805224e22a8758ea1ee3a8646" candidate_uuid: "eff2e38242d94402a1e5479ad01c8eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf2c2991983b405f9a967e56d29564e7"
I20260929 19:06:43.355535 26089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0300041805224e22a8758ea1ee3a8646" candidate_uuid: "eff2e38242d94402a1e5479ad01c8eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48b80b6f1c5446d79d93247980ac1f78"
I20260929 19:06:43.355599 26223 raft_consensus.cc:3037] T 0300041805224e22a8758ea1ee3a8646 P cf2c2991983b405f9a967e56d29564e7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:06:43.355623 26089 raft_consensus.cc:3037] T 0300041805224e22a8758ea1ee3a8646 P 48b80b6f1c5446d79d93247980ac1f78 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:06:43.356287 26223 raft_consensus.cc:2454] T 0300041805224e22a8758ea1ee3a8646 P cf2c2991983b405f9a967e56d29564e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff2e38242d94402a1e5479ad01c8eb3 in term 1.
I20260929 19:06:43.356307 26089 raft_consensus.cc:2454] T 0300041805224e22a8758ea1ee3a8646 P 48b80b6f1c5446d79d93247980ac1f78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff2e38242d94402a1e5479ad01c8eb3 in term 1.
I20260929 19:06:43.356488 25891 leader_election.cc:302] T 0300041805224e22a8758ea1ee3a8646 P eff2e38242d94402a1e5479ad01c8eb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf2c2991983b405f9a967e56d29564e7, eff2e38242d94402a1e5479ad01c8eb3; no voters:
I20260929 19:06:43.356572 26290 raft_consensus.cc:2781] T 0300041805224e22a8758ea1ee3a8646 P eff2e38242d94402a1e5479ad01c8eb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:06:43.356712 26290 raft_consensus.cc:686] T 0300041805224e22a8758ea1ee3a8646 P eff2e38242d94402a1e5479ad01c8eb3 [term 1 LEADER]: Becoming Leader. State: Replica: eff2e38242d94402a1e5479ad01c8eb3, State: Running, Role: LEADER
I20260929 19:06:43.356803 26290 consensus_queue.cc:237] T 0300041805224e22a8758ea1ee3a8646 P eff2e38242d94402a1e5479ad01c8eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b80b6f1c5446d79d93247980ac1f78" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 41907 } } peers { permanent_uuid: "cf2c2991983b405f9a967e56d29564e7" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45317 } } peers { permanent_uuid: "eff2e38242d94402a1e5479ad01c8eb3" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 42561 } }
I20260929 19:06:43.357923 25809 catalog_manager.cc:5495] T 0300041805224e22a8758ea1ee3a8646 P eff2e38242d94402a1e5479ad01c8eb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to eff2e38242d94402a1e5479ad01c8eb3 (127.8.158.65). New cstate: current_term: 1 leader_uuid: "eff2e38242d94402a1e5479ad01c8eb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b80b6f1c5446d79d93247980ac1f78" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 41907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf2c2991983b405f9a967e56d29564e7" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff2e38242d94402a1e5479ad01c8eb3" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 42561 } health_report { overall_health: HEALTHY } } }
I20260929 19:06:43.366139 26282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:43.363620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51956 (local address 127.8.158.65:42561)
0929 19:06:43.363678 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:43.363680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:43.363691 (+ 11us) server_negotiation.cc:407] Connection header received
0929 19:06:43.363726 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:43.363727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:43.363745 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:43.363814 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:06:43.363974 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.364452 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.365263 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.365502 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.365584 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:43.365769 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:06:43.365878 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:06:43.365970 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:43.366022 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 19:06:43.366042 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 19:06:43.367769 26223 raft_consensus.cc:1261] T 0300041805224e22a8758ea1ee3a8646 P cf2c2991983b405f9a967e56d29564e7 [term 1 FOLLOWER]: Refusing update from remote peer eff2e38242d94402a1e5479ad01c8eb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:06:43.367767 26089 raft_consensus.cc:1261] T 0300041805224e22a8758ea1ee3a8646 P 48b80b6f1c5446d79d93247980ac1f78 [term 1 FOLLOWER]: Refusing update from remote peer eff2e38242d94402a1e5479ad01c8eb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:06:43.368105 26290 consensus_queue.cc:1035] T 0300041805224e22a8758ea1ee3a8646 P eff2e38242d94402a1e5479ad01c8eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf2c2991983b405f9a967e56d29564e7" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:06:43.368245 26293 consensus_queue.cc:1035] T 0300041805224e22a8758ea1ee3a8646 P eff2e38242d94402a1e5479ad01c8eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48b80b6f1c5446d79d93247980ac1f78" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 41907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:06:43.370591 26298 mvcc.cc:205] Tried to move back new op lower bound from 7334743258591907840 to 7334743258550005760. Current Snapshot: MvccSnapshot[applied={T|T < 7334743258591907840}]
I20260929 19:06:43.371239 26297 mvcc.cc:205] Tried to move back new op lower bound from 7334743258591907840 to 7334743258550005760. Current Snapshot: MvccSnapshot[applied={T|T < 7334743258591907840}]
I20260929 19:06:43.372745 25810 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:06:43.372820 25810 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:06:43.373507 25810 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 42e37ba5a9134c168ae58e2676188068: Sending DeleteTablet for 3 replicas of tablet 0300041805224e22a8758ea1ee3a8646
I20260929 19:06:43.373852 26203 tablet_service.cc:1514] Processing DeleteTablet for tablet 0300041805224e22a8758ea1ee3a8646 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:06:43 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:33750
I20260929 19:06:43.373860 26069 tablet_service.cc:1514] Processing DeleteTablet for tablet 0300041805224e22a8758ea1ee3a8646 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:06:43 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:35544
I20260929 19:06:43.373883 25935 tablet_service.cc:1514] Processing DeleteTablet for tablet 0300041805224e22a8758ea1ee3a8646 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:06:43 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:51940
I20260929 19:06:43.374079 26308 tablet_replica.cc:330] T 0300041805224e22a8758ea1ee3a8646 P eff2e38242d94402a1e5479ad01c8eb3: stopping tablet replica
I20260929 19:06:43.374164 26308 raft_consensus.cc:2229] T 0300041805224e22a8758ea1ee3a8646 P eff2e38242d94402a1e5479ad01c8eb3 [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:06:43.374249 26308 raft_consensus.cc:2258] T 0300041805224e22a8758ea1ee3a8646 P eff2e38242d94402a1e5479ad01c8eb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:06:43.374415 26307 tablet_replica.cc:330] T 0300041805224e22a8758ea1ee3a8646 P 48b80b6f1c5446d79d93247980ac1f78: stopping tablet replica
I20260929 19:06:43.374420 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 25872
I20260929 19:06:43.374565 26307 raft_consensus.cc:2229] T 0300041805224e22a8758ea1ee3a8646 P 48b80b6f1c5446d79d93247980ac1f78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:06:43.374712 26307 raft_consensus.cc:2258] T 0300041805224e22a8758ea1ee3a8646 P 48b80b6f1c5446d79d93247980ac1f78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:06:43.374926 26306 tablet_replica.cc:330] T 0300041805224e22a8758ea1ee3a8646 P cf2c2991983b405f9a967e56d29564e7: stopping tablet replica
I20260929 19:06:43.375247 26306 raft_consensus.cc:2229] T 0300041805224e22a8758ea1ee3a8646 P cf2c2991983b405f9a967e56d29564e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:06:43.375396 26306 raft_consensus.cc:2258] T 0300041805224e22a8758ea1ee3a8646 P cf2c2991983b405f9a967e56d29564e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:06:43.375451 26307 ts_tablet_manager.cc:1905] T 0300041805224e22a8758ea1ee3a8646 P 48b80b6f1c5446d79d93247980ac1f78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:06:43.375957 26306 ts_tablet_manager.cc:1905] T 0300041805224e22a8758ea1ee3a8646 P cf2c2991983b405f9a967e56d29564e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:06:43.376832 26307 ts_tablet_manager.cc:1918] T 0300041805224e22a8758ea1ee3a8646 P 48b80b6f1c5446d79d93247980ac1f78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:06:43.376904 26307 log.cc:1196] T 0300041805224e22a8758ea1ee3a8646 P 48b80b6f1c5446d79d93247980ac1f78: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-1/wal/wals/0300041805224e22a8758ea1ee3a8646
I20260929 19:06:43.377102 26307 ts_tablet_manager.cc:1939] T 0300041805224e22a8758ea1ee3a8646 P 48b80b6f1c5446d79d93247980ac1f78: Deleting consensus metadata
I20260929 19:06:43.377415 25795 catalog_manager.cc:4841] TS 48b80b6f1c5446d79d93247980ac1f78 (127.8.158.66:41907): tablet 0300041805224e22a8758ea1ee3a8646 (table test-table [id=fe609663bb6e47caa2b32a6c4862a8f9]) successfully deleted
I20260929 19:06:43.378355 26306 ts_tablet_manager.cc:1918] T 0300041805224e22a8758ea1ee3a8646 P cf2c2991983b405f9a967e56d29564e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:06:43.378433 26306 log.cc:1196] T 0300041805224e22a8758ea1ee3a8646 P cf2c2991983b405f9a967e56d29564e7: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708573736807-8825-0/minicluster-data/ts-2/wal/wals/0300041805224e22a8758ea1ee3a8646
I20260929 19:06:43.378710 26306 ts_tablet_manager.cc:1939] T 0300041805224e22a8758ea1ee3a8646 P cf2c2991983b405f9a967e56d29564e7: Deleting consensus metadata
I20260929 19:06:43.379221 25797 catalog_manager.cc:4841] TS cf2c2991983b405f9a967e56d29564e7 (127.8.158.67:45317): tablet 0300041805224e22a8758ea1ee3a8646 (table test-table [id=fe609663bb6e47caa2b32a6c4862a8f9]) successfully deleted
W20260929 19:06:43.380690 25796 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.158.65:42561 (error 108)
W20260929 19:06:43.381924 26278 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:06:43.381051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:42561 (local address 127.0.0.1:51958)
0929 19:06:43.381100 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:43.381111 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:43.381147 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:43.381890 (+ 743us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.65:42561: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 19:06:43.382082 25796 catalog_manager.cc:4584] TS eff2e38242d94402a1e5479ad01c8eb3 (127.8.158.65:42561): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0300041805224e22a8758ea1ee3a8646: Network error: Client connection negotiation failed: client connection to 127.8.158.65:42561: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 19:06:43.382119 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 26006
I20260929 19:06:43.387627 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 26140
I20260929 19:06:43.392181 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 25778
2026-09-29T19:06:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1860 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T19:06:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:06:43Z Disabled control of system clock
I20260929 19:06:43.418874 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:39287
--webserver_interface=127.8.158.126
--webserver_port=0
--builtin_ntp_servers=127.8.158.84:41091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:39287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 19:06:43.497038 26313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:06:43.497243 26313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:06:43.497263 26313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:06:43.498435 26313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:06:43.498472 26313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:06:43.498485 26313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:06:43.498498 26313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:06:43.498510 26313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:06:43.499900 26313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:41091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:39287
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:39287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.158.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:06:43.500125 26313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:06:43.500430 26313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:06:43.505034 26321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:43.505026 26318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:43.505267 26313 server_base.cc:1034] running on GCE node
W20260929 19:06:43.505113 26319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:43.505606 26313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:06:43.505882 26313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:06:43.507066 26313 hybrid_clock.cc:648] HybridClock initialized: now 1790708803507040 us; error 47 us; skew 500 ppm
I20260929 19:06:43.508237 26313 webserver.cc:460] Webserver started at http://127.8.158.126:38189/ using document root <none> and password file <none>
I20260929 19:06:43.508482 26313 fs_manager.cc:359] Metadata directory not provided
I20260929 19:06:43.508549 26313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:06:43.508680 26313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:06:43.509666 26313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "e9ae7c90a7224a5bbe2c5b1ad7c6d100"
format_stamp: "Formatted at 2026-09-29 19:06:43 on dist-test-slave-nmsz"
server_key: "e596ead50f0135883dbb749a97e685b8"
server_key_iv: "85c78bd3556c08dd34650aeeb294a86f"
server_key_version: "encryptionkey@0"
I20260929 19:06:43.510007 26313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "e9ae7c90a7224a5bbe2c5b1ad7c6d100"
format_stamp: "Formatted at 2026-09-29 19:06:43 on dist-test-slave-nmsz"
server_key: "e596ead50f0135883dbb749a97e685b8"
server_key_iv: "85c78bd3556c08dd34650aeeb294a86f"
server_key_version: "encryptionkey@0"
I20260929 19:06:43.511288 26313 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:06:43.511963 26327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:43.512151 26313 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:06:43.512249 26313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "e9ae7c90a7224a5bbe2c5b1ad7c6d100"
format_stamp: "Formatted at 2026-09-29 19:06:43 on dist-test-slave-nmsz"
server_key: "e596ead50f0135883dbb749a97e685b8"
server_key_iv: "85c78bd3556c08dd34650aeeb294a86f"
server_key_version: "encryptionkey@0"
I20260929 19:06:43.512327 26313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:06:43.521761 26313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:06:43.522012 26313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:06:43.522158 26313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:06:43.525982 26313 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.126:39287
I20260929 19:06:43.526007 26379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.126:39287 every 8 connection(s)
I20260929 19:06:43.526417 26313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:06:43.527277 26380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:06:43.529310 26380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9ae7c90a7224a5bbe2c5b1ad7c6d100: Bootstrap starting.
I20260929 19:06:43.529798 26380 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9ae7c90a7224a5bbe2c5b1ad7c6d100: Neither blocks nor log segments found. Creating new log.
I20260929 19:06:43.530025 26380 log.cc:824] T 00000000000000000000000000000000 P e9ae7c90a7224a5bbe2c5b1ad7c6d100: Log is configured to *not* fsync() on all Append() calls
I20260929 19:06:43.530726 26380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9ae7c90a7224a5bbe2c5b1ad7c6d100: No bootstrap required, opened a new log
I20260929 19:06:43.532183 26380 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9ae7c90a7224a5bbe2c5b1ad7c6d100 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9ae7c90a7224a5bbe2c5b1ad7c6d100" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 39287 } }
I20260929 19:06:43.532307 26380 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9ae7c90a7224a5bbe2c5b1ad7c6d100 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:06:43.532330 26380 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9ae7c90a7224a5bbe2c5b1ad7c6d100 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9ae7c90a7224a5bbe2c5b1ad7c6d100, State: Initialized, Role: FOLLOWER
I20260929 19:06:43.532424 26380 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9ae7c90a7224a5bbe2c5b1ad7c6d100 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9ae7c90a7224a5bbe2c5b1ad7c6d100" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 39287 } }
I20260929 19:06:43.532502 26380 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9ae7c90a7224a5bbe2c5b1ad7c6d100 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:06:43.532537 26380 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9ae7c90a7224a5bbe2c5b1ad7c6d100 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:06:43.532603 26380 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9ae7c90a7224a5bbe2c5b1ad7c6d100 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:06:43.532709 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 26313
I20260929 19:06:43.532805 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:06:43.532992 8825 external_mini_cluster.cc:1428] Setting key cfbcc0ff252b1fa217915eb0bdccaf92
I20260929 19:06:43.533321 26380 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9ae7c90a7224a5bbe2c5b1ad7c6d100 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9ae7c90a7224a5bbe2c5b1ad7c6d100" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 39287 } }
I20260929 19:06:43.533420 26380 leader_election.cc:302] T 00000000000000000000000000000000 P e9ae7c90a7224a5bbe2c5b1ad7c6d100 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9ae7c90a7224a5bbe2c5b1ad7c6d100; no voters:
I20260929 19:06:43.533577 26380 leader_election.cc:288] T 00000000000000000000000000000000 P e9ae7c90a7224a5bbe2c5b1ad7c6d100 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:06:43.533716 26385 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9ae7c90a7224a5bbe2c5b1ad7c6d100 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:06:43.533829 26385 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9ae7c90a7224a5bbe2c5b1ad7c6d100 [term 1 LEADER]: Becoming Leader. State: Replica: e9ae7c90a7224a5bbe2c5b1ad7c6d100, State: Running, Role: LEADER
I20260929 19:06:43.533851 26380 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9ae7c90a7224a5bbe2c5b1ad7c6d100 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:06:43.533908 26385 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9ae7c90a7224a5bbe2c5b1ad7c6d100 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9ae7c90a7224a5bbe2c5b1ad7c6d100" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 39287 } }
I20260929 19:06:43.535308 26387 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9ae7c90a7224a5bbe2c5b1ad7c6d100 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9ae7c90a7224a5bbe2c5b1ad7c6d100. Latest consensus state: current_term: 1 leader_uuid: "e9ae7c90a7224a5bbe2c5b1ad7c6d100" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9ae7c90a7224a5bbe2c5b1ad7c6d100" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 39287 } } }
I20260929 19:06:43.535391 26387 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9ae7c90a7224a5bbe2c5b1ad7c6d100 [sys.catalog]: This master's current role is: LEADER
I20260929 19:06:43.535347 26386 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9ae7c90a7224a5bbe2c5b1ad7c6d100 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9ae7c90a7224a5bbe2c5b1ad7c6d100" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9ae7c90a7224a5bbe2c5b1ad7c6d100" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 39287 } } }
I20260929 19:06:43.535426 26386 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9ae7c90a7224a5bbe2c5b1ad7c6d100 [sys.catalog]: This master's current role is: LEADER
W20260929 19:06:43.535780 26400 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e9ae7c90a7224a5bbe2c5b1ad7c6d100: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 19:06:43.535828 26400 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:06:43.535984 26401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:06:43.536156 26401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:06:43.538689 26401 catalog_manager.cc:1347] Generated new cluster ID: e967496b73cc447fb03a42a5383df226
I20260929 19:06:43.538741 26401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:06:43.539014 26384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:43.533441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56848 (local address 127.8.158.126:39287)
0929 19:06:43.533669 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:43.533677 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:43.533709 (+ 32us) server_negotiation.cc:407] Connection header received
0929 19:06:43.533823 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:43.533830 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:43.533894 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:43.534032 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:43.534300 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.535666 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.536748 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.537297 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.537415 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:43.538081 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:43.538084 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:43.538086 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:06:43.538276 (+ 190us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:43.538339 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:06:43.538356 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:43.538444 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:43.538738 (+ 294us) server_negotiation.cc:299] Negotiation successful
0929 19:06:43.538812 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":106,"threads_started":1}
I20260929 19:06:43.543162 26401 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:06:43.543628 26401 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:06:43.545835 26401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9ae7c90a7224a5bbe2c5b1ad7c6d100: Generated new TSK 0
I20260929 19:06:43.545989 26401 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:06:43.548557 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.65:0
--local_ip_for_outbound_sockets=127.8.158.65
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:39287
--builtin_ntp_servers=127.8.158.84:41091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 19:06:43.640318 26404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:06:43.640501 26404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:06:43.640522 26404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:06:43.641646 26404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:06:43.641685 26404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:06:43.641711 26404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.65
I20260929 19:06:43.643148 26404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:41091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:39287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.65
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:06:43.643378 26404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:06:43.643672 26404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:06:43.648447 26409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:43.648468 26412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:43.648458 26410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:43.648844 26404 server_base.cc:1034] running on GCE node
I20260929 19:06:43.649036 26404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:06:43.649289 26404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:06:43.650452 26404 hybrid_clock.cc:648] HybridClock initialized: now 1790708803650439 us; error 30 us; skew 500 ppm
I20260929 19:06:43.651680 26404 webserver.cc:460] Webserver started at http://127.8.158.65:40679/ using document root <none> and password file <none>
I20260929 19:06:43.651929 26404 fs_manager.cc:359] Metadata directory not provided
I20260929 19:06:43.651997 26404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:06:43.652144 26404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:06:43.653177 26404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-0/data/instance:
uuid: "51849e0a8c084c9691b4c37720692b91"
format_stamp: "Formatted at 2026-09-29 19:06:43 on dist-test-slave-nmsz"
server_key: "70a9234bcea4ad9708153fcc67126eac"
server_key_iv: "984a87f4034c9797eeaead905e904e2d"
server_key_version: "encryptionkey@0"
I20260929 19:06:43.653524 26404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance:
uuid: "51849e0a8c084c9691b4c37720692b91"
format_stamp: "Formatted at 2026-09-29 19:06:43 on dist-test-slave-nmsz"
server_key: "70a9234bcea4ad9708153fcc67126eac"
server_key_iv: "984a87f4034c9797eeaead905e904e2d"
server_key_version: "encryptionkey@0"
I20260929 19:06:43.654927 26404 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:06:43.655761 26418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:43.656003 26404 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:06:43.656098 26404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-0/wal
uuid: "51849e0a8c084c9691b4c37720692b91"
format_stamp: "Formatted at 2026-09-29 19:06:43 on dist-test-slave-nmsz"
server_key: "70a9234bcea4ad9708153fcc67126eac"
server_key_iv: "984a87f4034c9797eeaead905e904e2d"
server_key_version: "encryptionkey@0"
I20260929 19:06:43.656170 26404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:06:43.679869 26404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:06:43.680217 26404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:06:43.680347 26404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:06:43.680572 26404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:06:43.681048 26404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:06:43.681114 26404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:43.681146 26404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:06:43.681164 26404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:43.688174 26404 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.65:37499
I20260929 19:06:43.688257 26531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.65:37499 every 8 connection(s)
I20260929 19:06:43.688573 26404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
I20260929 19:06:43.693739 26384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:43.689365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:49467 (local address 127.8.158.126:39287)
0929 19:06:43.689433 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:43.689435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:43.689722 (+ 287us) server_negotiation.cc:407] Connection header received
0929 19:06:43.690186 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:43.690187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:43.690198 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:43.690267 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:43.690966 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.691441 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.692383 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.692592 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.692664 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:43.693421 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:43.693422 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:43.693423 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:06:43.693427 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:43.693439 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:06:43.693443 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:43.693493 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:43.693580 (+ 87us) server_negotiation.cc:299] Negotiation successful
0929 19:06:43.693606 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 19:06:43.693822 26534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:43.689371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:39287 (local address 127.8.158.65:49467)
0929 19:06:43.689564 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:43.689606 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:43.690076 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:43.690376 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:43.690380 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:43.690392 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:06:43.690883 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:43.690887 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.691522 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.691523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:43.692276 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:43.692278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.692678 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.692679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:43.692779 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:43.693273 (+ 494us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:06:43.693274 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:06:43.693299 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:06:43.693319 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:06:43.693505 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:06:43.693509 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:06:43.693511 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 19:06:43.693561 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 19:06:43.693648 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
I20260929 19:06:43.693827 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 26404
I20260929 19:06:43.693909 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance
I20260929 19:06:43.694144 8825 external_mini_cluster.cc:1428] Setting key 5a830961e48e87bd223f15e64d384486
I20260929 19:06:43.694365 26532 heartbeater.cc:347] Connected to a master server at 127.8.158.126:39287
I20260929 19:06:43.694471 26532 heartbeater.cc:464] Registering TS with master...
I20260929 19:06:43.694648 26532 heartbeater.cc:511] Master 127.8.158.126:39287 requested a full tablet report, sending...
I20260929 19:06:43.695190 26344 ts_manager.cc:194] Registered new tserver with Master: 51849e0a8c084c9691b4c37720692b91 (127.8.158.65:37499)
I20260929 19:06:43.695211 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.66:0
--local_ip_for_outbound_sockets=127.8.158.66
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:39287
--builtin_ntp_servers=127.8.158.84:41091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 19:06:43.695931 26344 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.158.65:49467
W20260929 19:06:43.774732 26535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:06:43.774917 26535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:06:43.774938 26535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:06:43.776069 26535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:06:43.776108 26535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:06:43.776134 26535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.66
I20260929 19:06:43.777452 26535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:41091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:39287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.66
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:06:43.777689 26535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:06:43.777963 26535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:06:43.782701 26540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:43.782756 26543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:43.782811 26541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:43.782850 26535 server_base.cc:1034] running on GCE node
I20260929 19:06:43.783188 26535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:06:43.783501 26535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:06:43.784694 26535 hybrid_clock.cc:648] HybridClock initialized: now 1790708803784680 us; error 27 us; skew 500 ppm
I20260929 19:06:43.786000 26535 webserver.cc:460] Webserver started at http://127.8.158.66:45897/ using document root <none> and password file <none>
I20260929 19:06:43.786234 26535 fs_manager.cc:359] Metadata directory not provided
I20260929 19:06:43.786305 26535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:06:43.786484 26535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:06:43.787494 26535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-1/data/instance:
uuid: "19598befe09f45fe95b1e777267b84c3"
format_stamp: "Formatted at 2026-09-29 19:06:43 on dist-test-slave-nmsz"
server_key: "537758aafe3eb3b2fc97807bf493521e"
server_key_iv: "f64899af5db85a8055a77b2e12417ae2"
server_key_version: "encryptionkey@0"
I20260929 19:06:43.787839 26535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance:
uuid: "19598befe09f45fe95b1e777267b84c3"
format_stamp: "Formatted at 2026-09-29 19:06:43 on dist-test-slave-nmsz"
server_key: "537758aafe3eb3b2fc97807bf493521e"
server_key_iv: "f64899af5db85a8055a77b2e12417ae2"
server_key_version: "encryptionkey@0"
I20260929 19:06:43.789126 26535 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:06:43.789954 26549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:43.790246 26535 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:06:43.790357 26535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-1/wal
uuid: "19598befe09f45fe95b1e777267b84c3"
format_stamp: "Formatted at 2026-09-29 19:06:43 on dist-test-slave-nmsz"
server_key: "537758aafe3eb3b2fc97807bf493521e"
server_key_iv: "f64899af5db85a8055a77b2e12417ae2"
server_key_version: "encryptionkey@0"
I20260929 19:06:43.790444 26535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:06:43.825956 26535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:06:43.826272 26535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:06:43.826435 26535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:06:43.826663 26535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:06:43.826979 26535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:06:43.827032 26535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:43.827073 26535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:06:43.827126 26535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:43.832806 26535 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.66:33317
I20260929 19:06:43.832850 26662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.66:33317 every 8 connection(s)
I20260929 19:06:43.833158 26535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
I20260929 19:06:43.838838 26384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:43.833846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:57407 (local address 127.8.158.126:39287)
0929 19:06:43.833900 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:43.833902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:43.834161 (+ 259us) server_negotiation.cc:407] Connection header received
0929 19:06:43.834869 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:43.834870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:43.834883 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:43.834946 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:43.835689 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.836239 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.837309 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.837546 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.837634 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:43.838514 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:43.838516 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:43.838517 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:06:43.838521 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:43.838538 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:06:43.838550 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:43.838627 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:43.838725 (+ 98us) server_negotiation.cc:299] Negotiation successful
0929 19:06:43.838755 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 19:06:43.839061 26665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:43.833802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:39287 (local address 127.8.158.66:57407)
0929 19:06:43.834028 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:43.834062 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:43.834742 (+ 680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:43.835032 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:43.835036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:43.835046 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:06:43.835567 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:43.835572 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.836345 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.836346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:43.837191 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:43.837193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.837641 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.837642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:43.837791 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:43.838286 (+ 495us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:06:43.838287 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:06:43.838359 (+ 72us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:06:43.838408 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:06:43.838632 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:06:43.838638 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:06:43.838642 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:06:43.838720 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 19:06:43.838823 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":109,"threads_started":1}
I20260929 19:06:43.839537 26663 heartbeater.cc:347] Connected to a master server at 127.8.158.126:39287
I20260929 19:06:43.839630 26663 heartbeater.cc:464] Registering TS with master...
I20260929 19:06:43.839785 26663 heartbeater.cc:511] Master 127.8.158.126:39287 requested a full tablet report, sending...
I20260929 19:06:43.840004 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 26535
I20260929 19:06:43.840101 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance
I20260929 19:06:43.840157 26344 ts_manager.cc:194] Registered new tserver with Master: 19598befe09f45fe95b1e777267b84c3 (127.8.158.66:33317)
I20260929 19:06:43.840327 8825 external_mini_cluster.cc:1428] Setting key 795d7280d4149998d6bdaa51deb97834
I20260929 19:06:43.840564 26344 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.158.66:57407
I20260929 19:06:43.841310 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.67:0
--local_ip_for_outbound_sockets=127.8.158.67
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:39287
--builtin_ntp_servers=127.8.158.84:41091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 19:06:43.919297 26666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:06:43.919502 26666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:06:43.919535 26666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:06:43.920620 26666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:06:43.920680 26666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:06:43.920723 26666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.67
I20260929 19:06:43.922026 26666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:41091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:39287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.67
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:06:43.922271 26666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:06:43.922566 26666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:06:43.927238 26674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:43.927244 26671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:43.927306 26666 server_base.cc:1034] running on GCE node
W20260929 19:06:43.927249 26672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:43.927644 26666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:06:43.927874 26666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:06:43.929031 26666 hybrid_clock.cc:648] HybridClock initialized: now 1790708803929021 us; error 42 us; skew 500 ppm
I20260929 19:06:43.930248 26666 webserver.cc:460] Webserver started at http://127.8.158.67:44117/ using document root <none> and password file <none>
I20260929 19:06:43.930512 26666 fs_manager.cc:359] Metadata directory not provided
I20260929 19:06:43.930585 26666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:06:43.930732 26666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:06:43.931746 26666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-2/data/instance:
uuid: "86dd53da9f3b414287a5b7fd869e6739"
format_stamp: "Formatted at 2026-09-29 19:06:43 on dist-test-slave-nmsz"
server_key: "540713cf957b78430171715dbc2b27f0"
server_key_iv: "395e935a25367c1c4cdc4903c25363ac"
server_key_version: "encryptionkey@0"
I20260929 19:06:43.932080 26666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance:
uuid: "86dd53da9f3b414287a5b7fd869e6739"
format_stamp: "Formatted at 2026-09-29 19:06:43 on dist-test-slave-nmsz"
server_key: "540713cf957b78430171715dbc2b27f0"
server_key_iv: "395e935a25367c1c4cdc4903c25363ac"
server_key_version: "encryptionkey@0"
I20260929 19:06:43.933446 26666 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:06:43.934234 26680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:43.934449 26666 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:06:43.934542 26666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-2/wal
uuid: "86dd53da9f3b414287a5b7fd869e6739"
format_stamp: "Formatted at 2026-09-29 19:06:43 on dist-test-slave-nmsz"
server_key: "540713cf957b78430171715dbc2b27f0"
server_key_iv: "395e935a25367c1c4cdc4903c25363ac"
server_key_version: "encryptionkey@0"
I20260929 19:06:43.934628 26666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:06:43.945425 26666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:06:43.945675 26666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:06:43.945812 26666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:06:43.946023 26666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:06:43.946386 26666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:06:43.946447 26666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:43.946484 26666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:06:43.946511 26666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:43.952142 26666 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.67:42759
I20260929 19:06:43.952180 26793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.67:42759 every 8 connection(s)
I20260929 19:06:43.952483 26666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
I20260929 19:06:43.955698 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 26666
I20260929 19:06:43.955768 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance
I20260929 19:06:43.955960 8825 external_mini_cluster.cc:1428] Setting key 7e2d39e5bf5152692b5b5b7796010dda
I20260929 19:06:43.957613 26384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:43.953211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:48585 (local address 127.8.158.126:39287)
0929 19:06:43.953264 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:43.953265 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:43.953474 (+ 209us) server_negotiation.cc:407] Connection header received
0929 19:06:43.953919 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:43.953921 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:43.953932 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:43.953988 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:43.954681 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.955172 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.956245 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.956556 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.956620 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:43.957390 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:43.957392 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:43.957392 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:06:43.957404 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:43.957419 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:06:43.957423 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:43.957472 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:43.957534 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 19:06:43.957559 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 19:06:43.957849 26796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:43.953187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:39287 (local address 127.8.158.67:48585)
0929 19:06:43.953347 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:43.953383 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:43.953809 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:43.954034 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:43.954037 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:43.954045 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:06:43.954594 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:43.954599 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.955248 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.955249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:43.956125 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:43.956128 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.956644 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.956645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:43.956792 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:43.957257 (+ 465us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:06:43.957257 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:06:43.957281 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:06:43.957301 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:06:43.957483 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:06:43.957487 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:06:43.957489 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 19:06:43.957536 (+ 47us) client_negotiation.cc:240] Negotiation successful
0929 19:06:43.957607 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":78,"threads_started":1}
I20260929 19:06:43.958401 26794 heartbeater.cc:347] Connected to a master server at 127.8.158.126:39287
I20260929 19:06:43.958508 26794 heartbeater.cc:464] Registering TS with master...
I20260929 19:06:43.958686 26794 heartbeater.cc:511] Master 127.8.158.126:39287 requested a full tablet report, sending...
I20260929 19:06:43.959051 26343 ts_manager.cc:194] Registered new tserver with Master: 86dd53da9f3b414287a5b7fd869e6739 (127.8.158.67:42759)
I20260929 19:06:43.959406 26343 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.158.67:48585
I20260929 19:06:43.967695 8825 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:06:43.972326 26384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:43.969141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56858 (local address 127.8.158.126:39287)
0929 19:06:43.969236 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:43.969238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:43.969348 (+ 110us) server_negotiation.cc:407] Connection header received
0929 19:06:43.969419 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:43.969420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:43.969433 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:43.969512 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:43.969701 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.970212 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.971064 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.971269 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.971361 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:43.972078 (+ 717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:43.972080 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:43.972080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:06:43.972084 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:43.972098 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:06:43.972102 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:43.972171 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:43.972225 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 19:06:43.972247 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 19:06:43.974020 26343 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56858:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:06:43.975602 26343 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:06:43.982007 26802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:43.977762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.67:42759 (local address 127.0.0.1:43312)
0929 19:06:43.977872 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:43.977909 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:43.977966 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:43.978565 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:43.978568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:43.978571 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:06:43.978633 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:43.978635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.979832 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.979833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:43.981098 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:43.981099 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.981738 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.981739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:43.981809 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:43.981821 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 19:06:43.981890 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 19:06:43.981937 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":4,"thread_start_us":66,"threads_started":1}
I20260929 19:06:43.982015 26804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:43.977932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43312 (local address 127.8.158.67:42759)
0929 19:06:43.978283 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:43.978287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:43.978296 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:06:43.978392 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:43.978394 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:43.978407 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:43.978520 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:06:43.978776 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.979732 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.981197 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.981654 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.981737 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:43.981821 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:43.981901 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 19:06:43.981933 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":67,"threads_started":1}
I20260929 19:06:43.982209 26805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:43.977834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.66:33317 (local address 127.0.0.1:51454)
0929 19:06:43.978096 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:43.978106 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:43.978151 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:43.978949 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:43.978950 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:43.978951 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:06:43.979006 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:43.979007 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.980231 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.980233 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:43.981112 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:43.981115 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.981983 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.981984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:43.982054 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:43.982059 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 19:06:43.982092 (+ 33us) client_negotiation.cc:240] Negotiation successful
0929 19:06:43.982134 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":87,"threads_started":1}
I20260929 19:06:43.982491 26803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:43.977968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:37499 (local address 127.0.0.1:44550)
0929 19:06:43.978117 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:43.978126 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:43.978165 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:43.978686 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:43.978687 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:43.978690 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:06:43.978737 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:43.978738 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.980026 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.980027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:43.981095 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:43.981098 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.982197 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.982198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:43.982256 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:43.982260 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:06:43.982381 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 19:06:43.982417 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":84,"threads_started":1}
I20260929 19:06:43.982600 26806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:43.978082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51454 (local address 127.8.158.66:33317)
0929 19:06:43.978438 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:43.978450 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:43.978717 (+ 267us) server_negotiation.cc:407] Connection header received
0929 19:06:43.978768 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:43.978770 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:43.978787 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:43.978937 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:06:43.979098 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.980152 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.981220 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.981892 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.982343 (+ 451us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:43.982432 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:43.982456 (+ 24us) server_negotiation.cc:299] Negotiation successful
0929 19:06:43.982495 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":73,"threads_started":1}
I20260929 19:06:43.982712 26728 tablet_service.cc:1467] Processing CreateTablet for tablet 474e41ec450440a495ec246041929d19 (DEFAULT_TABLE table=test-table [id=fb2f56f5855c4e41a1e601281f6bc136]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:06:43.982817 26807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:43.978076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44550 (local address 127.8.158.65:37499)
0929 19:06:43.978293 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:43.978296 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:43.978304 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:06:43.978386 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:43.978388 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:43.978404 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:43.978534 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:06:43.978835 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.979922 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.981187 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:43.981880 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:43.981954 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:43.982034 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:43.982647 (+ 613us) server_negotiation.cc:299] Negotiation successful
0929 19:06:43.982692 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":84,"threads_started":1}
I20260929 19:06:43.983062 26728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 474e41ec450440a495ec246041929d19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:06:43.983178 26597 tablet_service.cc:1467] Processing CreateTablet for tablet 474e41ec450440a495ec246041929d19 (DEFAULT_TABLE table=test-table [id=fb2f56f5855c4e41a1e601281f6bc136]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:06:43.983458 26597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 474e41ec450440a495ec246041929d19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:06:43.983407 26466 tablet_service.cc:1467] Processing CreateTablet for tablet 474e41ec450440a495ec246041929d19 (DEFAULT_TABLE table=test-table [id=fb2f56f5855c4e41a1e601281f6bc136]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:06:43.983633 26466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 474e41ec450440a495ec246041929d19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:06:43.986208 26808 tablet_bootstrap.cc:492] T 474e41ec450440a495ec246041929d19 P 51849e0a8c084c9691b4c37720692b91: Bootstrap starting.
I20260929 19:06:43.986378 26809 tablet_bootstrap.cc:492] T 474e41ec450440a495ec246041929d19 P 86dd53da9f3b414287a5b7fd869e6739: Bootstrap starting.
I20260929 19:06:43.986881 26810 tablet_bootstrap.cc:492] T 474e41ec450440a495ec246041929d19 P 19598befe09f45fe95b1e777267b84c3: Bootstrap starting.
I20260929 19:06:43.986927 26808 tablet_bootstrap.cc:654] T 474e41ec450440a495ec246041929d19 P 51849e0a8c084c9691b4c37720692b91: Neither blocks nor log segments found. Creating new log.
I20260929 19:06:43.986927 26809 tablet_bootstrap.cc:654] T 474e41ec450440a495ec246041929d19 P 86dd53da9f3b414287a5b7fd869e6739: Neither blocks nor log segments found. Creating new log.
I20260929 19:06:43.987246 26809 log.cc:824] T 474e41ec450440a495ec246041929d19 P 86dd53da9f3b414287a5b7fd869e6739: Log is configured to *not* fsync() on all Append() calls
I20260929 19:06:43.987246 26808 log.cc:824] T 474e41ec450440a495ec246041929d19 P 51849e0a8c084c9691b4c37720692b91: Log is configured to *not* fsync() on all Append() calls
I20260929 19:06:43.987498 26810 tablet_bootstrap.cc:654] T 474e41ec450440a495ec246041929d19 P 19598befe09f45fe95b1e777267b84c3: Neither blocks nor log segments found. Creating new log.
I20260929 19:06:43.987814 26810 log.cc:824] T 474e41ec450440a495ec246041929d19 P 19598befe09f45fe95b1e777267b84c3: Log is configured to *not* fsync() on all Append() calls
I20260929 19:06:43.987957 26809 tablet_bootstrap.cc:492] T 474e41ec450440a495ec246041929d19 P 86dd53da9f3b414287a5b7fd869e6739: No bootstrap required, opened a new log
I20260929 19:06:43.988041 26809 ts_tablet_manager.cc:1397] T 474e41ec450440a495ec246041929d19 P 86dd53da9f3b414287a5b7fd869e6739: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 19:06:43.988194 26808 tablet_bootstrap.cc:492] T 474e41ec450440a495ec246041929d19 P 51849e0a8c084c9691b4c37720692b91: No bootstrap required, opened a new log
I20260929 19:06:43.988260 26808 ts_tablet_manager.cc:1397] T 474e41ec450440a495ec246041929d19 P 51849e0a8c084c9691b4c37720692b91: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:06:43.988540 26810 tablet_bootstrap.cc:492] T 474e41ec450440a495ec246041929d19 P 19598befe09f45fe95b1e777267b84c3: No bootstrap required, opened a new log
I20260929 19:06:43.988601 26810 ts_tablet_manager.cc:1397] T 474e41ec450440a495ec246041929d19 P 19598befe09f45fe95b1e777267b84c3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:06:43.990259 26808 raft_consensus.cc:348] T 474e41ec450440a495ec246041929d19 P 51849e0a8c084c9691b4c37720692b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86dd53da9f3b414287a5b7fd869e6739" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 42759 } } peers { permanent_uuid: "51849e0a8c084c9691b4c37720692b91" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 37499 } } peers { permanent_uuid: "19598befe09f45fe95b1e777267b84c3" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 33317 } }
I20260929 19:06:43.990447 26808 raft_consensus.cc:374] T 474e41ec450440a495ec246041929d19 P 51849e0a8c084c9691b4c37720692b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:06:43.990501 26808 raft_consensus.cc:729] T 474e41ec450440a495ec246041929d19 P 51849e0a8c084c9691b4c37720692b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51849e0a8c084c9691b4c37720692b91, State: Initialized, Role: FOLLOWER
I20260929 19:06:43.990634 26808 consensus_queue.cc:260] T 474e41ec450440a495ec246041929d19 P 51849e0a8c084c9691b4c37720692b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86dd53da9f3b414287a5b7fd869e6739" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 42759 } } peers { permanent_uuid: "51849e0a8c084c9691b4c37720692b91" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 37499 } } peers { permanent_uuid: "19598befe09f45fe95b1e777267b84c3" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 33317 } }
I20260929 19:06:43.990893 26808 ts_tablet_manager.cc:1428] T 474e41ec450440a495ec246041929d19 P 51849e0a8c084c9691b4c37720692b91: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 19:06:43.990830 26809 raft_consensus.cc:348] T 474e41ec450440a495ec246041929d19 P 86dd53da9f3b414287a5b7fd869e6739 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86dd53da9f3b414287a5b7fd869e6739" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 42759 } } peers { permanent_uuid: "51849e0a8c084c9691b4c37720692b91" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 37499 } } peers { permanent_uuid: "19598befe09f45fe95b1e777267b84c3" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 33317 } }
I20260929 19:06:43.991034 26532 heartbeater.cc:503] Master 127.8.158.126:39287 was elected leader, sending a full tablet report...
I20260929 19:06:43.991276 26809 raft_consensus.cc:374] T 474e41ec450440a495ec246041929d19 P 86dd53da9f3b414287a5b7fd869e6739 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:06:43.991339 26809 raft_consensus.cc:729] T 474e41ec450440a495ec246041929d19 P 86dd53da9f3b414287a5b7fd869e6739 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86dd53da9f3b414287a5b7fd869e6739, State: Initialized, Role: FOLLOWER
I20260929 19:06:43.991370 26810 raft_consensus.cc:348] T 474e41ec450440a495ec246041929d19 P 19598befe09f45fe95b1e777267b84c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86dd53da9f3b414287a5b7fd869e6739" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 42759 } } peers { permanent_uuid: "51849e0a8c084c9691b4c37720692b91" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 37499 } } peers { permanent_uuid: "19598befe09f45fe95b1e777267b84c3" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 33317 } }
I20260929 19:06:43.991460 26809 consensus_queue.cc:260] T 474e41ec450440a495ec246041929d19 P 86dd53da9f3b414287a5b7fd869e6739 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86dd53da9f3b414287a5b7fd869e6739" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 42759 } } peers { permanent_uuid: "51849e0a8c084c9691b4c37720692b91" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 37499 } } peers { permanent_uuid: "19598befe09f45fe95b1e777267b84c3" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 33317 } }
I20260929 19:06:43.991513 26810 raft_consensus.cc:374] T 474e41ec450440a495ec246041929d19 P 19598befe09f45fe95b1e777267b84c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:06:43.991556 26810 raft_consensus.cc:729] T 474e41ec450440a495ec246041929d19 P 19598befe09f45fe95b1e777267b84c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19598befe09f45fe95b1e777267b84c3, State: Initialized, Role: FOLLOWER
I20260929 19:06:43.991696 26809 ts_tablet_manager.cc:1428] T 474e41ec450440a495ec246041929d19 P 86dd53da9f3b414287a5b7fd869e6739: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260929 19:06:43.991707 26810 consensus_queue.cc:260] T 474e41ec450440a495ec246041929d19 P 19598befe09f45fe95b1e777267b84c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86dd53da9f3b414287a5b7fd869e6739" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 42759 } } peers { permanent_uuid: "51849e0a8c084c9691b4c37720692b91" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 37499 } } peers { permanent_uuid: "19598befe09f45fe95b1e777267b84c3" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 33317 } }
I20260929 19:06:43.991804 26794 heartbeater.cc:503] Master 127.8.158.126:39287 was elected leader, sending a full tablet report...
I20260929 19:06:43.991946 26810 ts_tablet_manager.cc:1428] T 474e41ec450440a495ec246041929d19 P 19598befe09f45fe95b1e777267b84c3: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260929 19:06:43.992025 26663 heartbeater.cc:503] Master 127.8.158.126:39287 was elected leader, sending a full tablet report...
I20260929 19:06:44.015110 26816 raft_consensus.cc:482] T 474e41ec450440a495ec246041929d19 P 19598befe09f45fe95b1e777267b84c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:06:44.015225 26816 raft_consensus.cc:504] T 474e41ec450440a495ec246041929d19 P 19598befe09f45fe95b1e777267b84c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86dd53da9f3b414287a5b7fd869e6739" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 42759 } } peers { permanent_uuid: "51849e0a8c084c9691b4c37720692b91" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 37499 } } peers { permanent_uuid: "19598befe09f45fe95b1e777267b84c3" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 33317 } }
I20260929 19:06:44.015569 26816 leader_election.cc:288] T 474e41ec450440a495ec246041929d19 P 19598befe09f45fe95b1e777267b84c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86dd53da9f3b414287a5b7fd869e6739 (127.8.158.67:42759), 51849e0a8c084c9691b4c37720692b91 (127.8.158.65:37499)
I20260929 19:06:44.018445 26665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:44.015710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.67:42759 (local address 127.8.158.66:40207)
0929 19:06:44.015747 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:44.015759 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:44.015800 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:44.015923 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:44.015924 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:44.015925 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:06:44.015996 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:44.015998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:44.016751 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:44.016752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:44.017573 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:44.017575 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:44.018155 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:44.018156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:44.018225 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:44.018228 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:06:44.018289 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 19:06:44.018349 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17,"mutex_wait_us":25}
I20260929 19:06:44.018445 26804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:44.015780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:40207 (local address 127.8.158.67:42759)
0929 19:06:44.015817 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:44.015820 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:44.015827 (+ 7us) server_negotiation.cc:407] Connection header received
0929 19:06:44.015856 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:44.015861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:44.015869 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:44.015915 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:06:44.016100 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:44.016676 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:44.017658 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:44.018076 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:44.018138 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:44.018145 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:44.018298 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 19:06:44.018375 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 19:06:44.018718 26807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:44.015994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:49627 (local address 127.8.158.65:37499)
0929 19:06:44.016062 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:44.016063 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:44.016073 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:06:44.016153 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:44.016154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:44.016162 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:44.016204 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:06:44.016360 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:44.016860 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:44.017867 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:44.018299 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:44.018400 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:44.018408 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:44.018620 (+ 212us) server_negotiation.cc:299] Negotiation successful
0929 19:06:44.018661 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 19:06:44.018715 26817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:44.015816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:37499 (local address 127.8.158.66:49627)
0929 19:06:44.016033 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:44.016043 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:44.016085 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:44.016228 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:44.016229 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:44.016230 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:06:44.016289 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:44.016291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:44.016935 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:44.016936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:44.017784 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:44.017787 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:44.018421 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:44.018422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:44.018504 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:44.018508 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:06:44.018606 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 19:06:44.018643 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":89,"threads_started":1}
I20260929 19:06:44.018779 26748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "474e41ec450440a495ec246041929d19" candidate_uuid: "19598befe09f45fe95b1e777267b84c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86dd53da9f3b414287a5b7fd869e6739" is_pre_election: true
I20260929 19:06:44.018924 26748 raft_consensus.cc:2454] T 474e41ec450440a495ec246041929d19 P 86dd53da9f3b414287a5b7fd869e6739 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19598befe09f45fe95b1e777267b84c3 in term 0.
I20260929 19:06:44.019174 26550 leader_election.cc:302] T 474e41ec450440a495ec246041929d19 P 19598befe09f45fe95b1e777267b84c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19598befe09f45fe95b1e777267b84c3, 86dd53da9f3b414287a5b7fd869e6739; no voters:
I20260929 19:06:44.019248 26486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "474e41ec450440a495ec246041929d19" candidate_uuid: "19598befe09f45fe95b1e777267b84c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51849e0a8c084c9691b4c37720692b91" is_pre_election: true
I20260929 19:06:44.019304 26816 raft_consensus.cc:2781] T 474e41ec450440a495ec246041929d19 P 19598befe09f45fe95b1e777267b84c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:06:44.019369 26816 raft_consensus.cc:482] T 474e41ec450440a495ec246041929d19 P 19598befe09f45fe95b1e777267b84c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:06:44.019407 26816 raft_consensus.cc:3037] T 474e41ec450440a495ec246041929d19 P 19598befe09f45fe95b1e777267b84c3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:06:44.019395 26486 raft_consensus.cc:2454] T 474e41ec450440a495ec246041929d19 P 51849e0a8c084c9691b4c37720692b91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19598befe09f45fe95b1e777267b84c3 in term 0.
I20260929 19:06:44.020153 26816 raft_consensus.cc:504] T 474e41ec450440a495ec246041929d19 P 19598befe09f45fe95b1e777267b84c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86dd53da9f3b414287a5b7fd869e6739" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 42759 } } peers { permanent_uuid: "51849e0a8c084c9691b4c37720692b91" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 37499 } } peers { permanent_uuid: "19598befe09f45fe95b1e777267b84c3" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 33317 } }
I20260929 19:06:44.020280 26816 leader_election.cc:288] T 474e41ec450440a495ec246041929d19 P 19598befe09f45fe95b1e777267b84c3 [CANDIDATE]: Term 1 election: Requested vote from peers 86dd53da9f3b414287a5b7fd869e6739 (127.8.158.67:42759), 51849e0a8c084c9691b4c37720692b91 (127.8.158.65:37499)
I20260929 19:06:44.020471 26748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "474e41ec450440a495ec246041929d19" candidate_uuid: "19598befe09f45fe95b1e777267b84c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86dd53da9f3b414287a5b7fd869e6739"
I20260929 19:06:44.020545 26748 raft_consensus.cc:3037] T 474e41ec450440a495ec246041929d19 P 86dd53da9f3b414287a5b7fd869e6739 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:06:44.020541 26486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "474e41ec450440a495ec246041929d19" candidate_uuid: "19598befe09f45fe95b1e777267b84c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51849e0a8c084c9691b4c37720692b91"
I20260929 19:06:44.020640 26486 raft_consensus.cc:3037] T 474e41ec450440a495ec246041929d19 P 51849e0a8c084c9691b4c37720692b91 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:06:44.021198 26748 raft_consensus.cc:2454] T 474e41ec450440a495ec246041929d19 P 86dd53da9f3b414287a5b7fd869e6739 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19598befe09f45fe95b1e777267b84c3 in term 1.
I20260929 19:06:44.021363 26486 raft_consensus.cc:2454] T 474e41ec450440a495ec246041929d19 P 51849e0a8c084c9691b4c37720692b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19598befe09f45fe95b1e777267b84c3 in term 1.
I20260929 19:06:44.021387 26550 leader_election.cc:302] T 474e41ec450440a495ec246041929d19 P 19598befe09f45fe95b1e777267b84c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19598befe09f45fe95b1e777267b84c3, 86dd53da9f3b414287a5b7fd869e6739; no voters:
I20260929 19:06:44.021505 26816 raft_consensus.cc:2781] T 474e41ec450440a495ec246041929d19 P 19598befe09f45fe95b1e777267b84c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:06:44.021647 26816 raft_consensus.cc:686] T 474e41ec450440a495ec246041929d19 P 19598befe09f45fe95b1e777267b84c3 [term 1 LEADER]: Becoming Leader. State: Replica: 19598befe09f45fe95b1e777267b84c3, State: Running, Role: LEADER
I20260929 19:06:44.021766 26816 consensus_queue.cc:237] T 474e41ec450440a495ec246041929d19 P 19598befe09f45fe95b1e777267b84c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86dd53da9f3b414287a5b7fd869e6739" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 42759 } } peers { permanent_uuid: "51849e0a8c084c9691b4c37720692b91" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 37499 } } peers { permanent_uuid: "19598befe09f45fe95b1e777267b84c3" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 33317 } }
I20260929 19:06:44.022541 26344 catalog_manager.cc:5495] T 474e41ec450440a495ec246041929d19 P 19598befe09f45fe95b1e777267b84c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19598befe09f45fe95b1e777267b84c3 (127.8.158.66). New cstate: current_term: 1 leader_uuid: "19598befe09f45fe95b1e777267b84c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86dd53da9f3b414287a5b7fd869e6739" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 42759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51849e0a8c084c9691b4c37720692b91" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 37499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19598befe09f45fe95b1e777267b84c3" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 33317 } health_report { overall_health: HEALTHY } } }
I20260929 19:06:44.039448 26806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:44.036676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51456 (local address 127.8.158.66:33317)
0929 19:06:44.036720 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:44.036722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:44.036944 (+ 222us) server_negotiation.cc:407] Connection header received
0929 19:06:44.037018 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:44.037019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:44.037038 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:44.037130 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:06:44.037262 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:44.037710 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:44.038576 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:44.038858 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:44.038952 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:44.039129 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:06:44.039230 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:06:44.039320 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:44.039339 (+ 19us) server_negotiation.cc:299] Negotiation successful
0929 19:06:44.039358 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 19:06:44.040946 26486 raft_consensus.cc:1261] T 474e41ec450440a495ec246041929d19 P 51849e0a8c084c9691b4c37720692b91 [term 1 FOLLOWER]: Refusing update from remote peer 19598befe09f45fe95b1e777267b84c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:06:44.040951 26748 raft_consensus.cc:1261] T 474e41ec450440a495ec246041929d19 P 86dd53da9f3b414287a5b7fd869e6739 [term 1 FOLLOWER]: Refusing update from remote peer 19598befe09f45fe95b1e777267b84c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:06:44.041306 26818 consensus_queue.cc:1035] T 474e41ec450440a495ec246041929d19 P 19598befe09f45fe95b1e777267b84c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51849e0a8c084c9691b4c37720692b91" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 37499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:06:44.041437 26816 consensus_queue.cc:1035] T 474e41ec450440a495ec246041929d19 P 19598befe09f45fe95b1e777267b84c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86dd53da9f3b414287a5b7fd869e6739" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 42759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:06:44.043538 26821 mvcc.cc:205] Tried to move back new op lower bound from 7334743261349388288 to 7334743261273837568. Current Snapshot: MvccSnapshot[applied={T|T < 7334743261349388288}]
I20260929 19:06:44.043967 26823 mvcc.cc:205] Tried to move back new op lower bound from 7334743261349388288 to 7334743261273837568. Current Snapshot: MvccSnapshot[applied={T|T < 7334743261349388288}]
I20260929 19:06:44.045274 26822 mvcc.cc:205] Tried to move back new op lower bound from 7334743261349388288 to 7334743261273837568. Current Snapshot: MvccSnapshot[applied={T|T < 7334743261349388288}]
I20260929 19:06:44.045408 26343 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:06:44.045495 26343 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:06:44.046164 26343 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e9ae7c90a7224a5bbe2c5b1ad7c6d100: Sending DeleteTablet for 3 replicas of tablet 474e41ec450440a495ec246041929d19
I20260929 19:06:44.046505 26597 tablet_service.cc:1514] Processing DeleteTablet for tablet 474e41ec450440a495ec246041929d19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:06:44 UTC) from {username='slave'} at 127.0.0.1:51454
I20260929 19:06:44.046725 26831 tablet_replica.cc:330] T 474e41ec450440a495ec246041929d19 P 19598befe09f45fe95b1e777267b84c3: stopping tablet replica
I20260929 19:06:44.046726 26466 tablet_service.cc:1514] Processing DeleteTablet for tablet 474e41ec450440a495ec246041929d19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:06:44 UTC) from {username='slave'} at 127.0.0.1:44550
I20260929 19:06:44.046823 26831 raft_consensus.cc:2229] T 474e41ec450440a495ec246041929d19 P 19598befe09f45fe95b1e777267b84c3 [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:06:44.046918 26831 raft_consensus.cc:2258] T 474e41ec450440a495ec246041929d19 P 19598befe09f45fe95b1e777267b84c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:06:44.046988 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 26404
I20260929 19:06:44.046986 26832 tablet_replica.cc:330] T 474e41ec450440a495ec246041929d19 P 51849e0a8c084c9691b4c37720692b91: stopping tablet replica
I20260929 19:06:44.047129 26832 raft_consensus.cc:2229] T 474e41ec450440a495ec246041929d19 P 51849e0a8c084c9691b4c37720692b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:06:44.047276 26832 raft_consensus.cc:2258] T 474e41ec450440a495ec246041929d19 P 51849e0a8c084c9691b4c37720692b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:06:44.047966 26831 ts_tablet_manager.cc:1905] T 474e41ec450440a495ec246041929d19 P 19598befe09f45fe95b1e777267b84c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:06:44.049922 26831 ts_tablet_manager.cc:1918] T 474e41ec450440a495ec246041929d19 P 19598befe09f45fe95b1e777267b84c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:06:44.049996 26831 log.cc:1196] T 474e41ec450440a495ec246041929d19 P 19598befe09f45fe95b1e777267b84c3: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-1/wal/wals/474e41ec450440a495ec246041929d19
I20260929 19:06:44.050248 26831 ts_tablet_manager.cc:1939] T 474e41ec450440a495ec246041929d19 P 19598befe09f45fe95b1e777267b84c3: Deleting consensus metadata
I20260929 19:06:44.050823 26728 tablet_service.cc:1514] Processing DeleteTablet for tablet 474e41ec450440a495ec246041929d19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:06:44 UTC) from {username='slave'} at 127.0.0.1:43312
I20260929 19:06:44.050913 26331 catalog_manager.cc:4841] TS 19598befe09f45fe95b1e777267b84c3 (127.8.158.66:33317): tablet 474e41ec450440a495ec246041929d19 (table test-table [id=fb2f56f5855c4e41a1e601281f6bc136]) successfully deleted
I20260929 19:06:44.051047 26833 tablet_replica.cc:330] T 474e41ec450440a495ec246041929d19 P 86dd53da9f3b414287a5b7fd869e6739: stopping tablet replica
I20260929 19:06:44.051133 26833 raft_consensus.cc:2229] T 474e41ec450440a495ec246041929d19 P 86dd53da9f3b414287a5b7fd869e6739 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:06:44.051235 26833 raft_consensus.cc:2258] T 474e41ec450440a495ec246041929d19 P 86dd53da9f3b414287a5b7fd869e6739 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 19:06:44.054103 26328 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.158.65:37499 (error 108)
I20260929 19:06:44.054303 26833 ts_tablet_manager.cc:1905] T 474e41ec450440a495ec246041929d19 P 86dd53da9f3b414287a5b7fd869e6739: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:06:44.054375 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 26535
I20260929 19:06:44.055079 26803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:44.054602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:37499 (local address 127.0.0.1:44560)
0929 19:06:44.055011 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:44.055048 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.65:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393}
W20260929 19:06:44.055615 26328 catalog_manager.cc:4584] TS 51849e0a8c084c9691b4c37720692b91 (127.8.158.65:37499): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 474e41ec450440a495ec246041929d19: Network error: Client connection negotiation failed: client connection to 127.8.158.65:37499: connect: Connection refused (error 111)
I20260929 19:06:44.056811 26833 ts_tablet_manager.cc:1918] T 474e41ec450440a495ec246041929d19 P 86dd53da9f3b414287a5b7fd869e6739: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:06:44.056885 26833 log.cc:1196] T 474e41ec450440a495ec246041929d19 P 86dd53da9f3b414287a5b7fd869e6739: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708573736807-8825-0/minicluster-data/ts-2/wal/wals/474e41ec450440a495ec246041929d19
I20260929 19:06:44.057122 26833 ts_tablet_manager.cc:1939] T 474e41ec450440a495ec246041929d19 P 86dd53da9f3b414287a5b7fd869e6739: Deleting consensus metadata
I20260929 19:06:44.057577 26328 catalog_manager.cc:4841] TS 86dd53da9f3b414287a5b7fd869e6739 (127.8.158.67:42759): tablet 474e41ec450440a495ec246041929d19 (table test-table [id=fb2f56f5855c4e41a1e601281f6bc136]) successfully deleted
I20260929 19:06:44.060106 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 26666
I20260929 19:06:44.064709 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 26313
2026-09-29T19:06:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (671 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:06:44 dist-test-slave-nmsz krb5kdc[26837](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:06:44 dist-test-slave-nmsz krb5kdc[26837](info): set up 2 sockets
Sep 29 19:06:44 dist-test-slave-nmsz krb5kdc[26837](info): commencing operation
krb5kdc: starting...
W20260929 19:06:45.114099 8825 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Sep 29 19:06:45 dist-test-slave-nmsz krb5kdc[26837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:06:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:06:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.158.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
I20260929 19:06:45.225212 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:42691
--webserver_interface=127.8.158.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:40233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:42691
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:06:45.303537 26850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:06:45.303758 26850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:06:45.303799 26850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:06:45.304939 26850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:06:45.305008 26850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:06:45.305032 26850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:06:45.305060 26850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:06:45.305090 26850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:06:45.306511 26850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:40233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:42691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:42691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.158.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:06:45.306769 26850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:06:45.306979 26850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:06:45.311561 26858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:45.311609 26856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:45.311775 26855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:45.311695 26850 server_base.cc:1034] running on GCE node
I20260929 19:06:45.312000 26850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:06:45.312227 26850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:06:45.313362 26850 hybrid_clock.cc:648] HybridClock initialized: now 1790708805313349 us; error 29 us; skew 500 ppm
Sep 29 19:06:45 dist-test-slave-nmsz krb5kdc[26837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708805, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:06:45.315739 26850 init.cc:377] Logged in from keytab as kudu/127.8.158.126@KRBTEST.COM (short username kudu)
I20260929 19:06:45.316172 26850 webserver.cc:460] Webserver started at http://127.8.158.126:37695/ using document root <none> and password file <none>
I20260929 19:06:45.316354 26850 fs_manager.cc:359] Metadata directory not provided
I20260929 19:06:45.316406 26850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:06:45.316488 26850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:06:45.317456 26850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "db8afd05497843978227d3794bac6adf"
format_stamp: "Formatted at 2026-09-29 19:06:45 on dist-test-slave-nmsz"
server_key: "cf1f3967bb2058a93434c49e3f2190b1"
server_key_iv: "0421be70e966c83ef2e194a34be67e5d"
server_key_version: "encryptionkey@0"
I20260929 19:06:45.317736 26850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "db8afd05497843978227d3794bac6adf"
format_stamp: "Formatted at 2026-09-29 19:06:45 on dist-test-slave-nmsz"
server_key: "cf1f3967bb2058a93434c49e3f2190b1"
server_key_iv: "0421be70e966c83ef2e194a34be67e5d"
server_key_version: "encryptionkey@0"
I20260929 19:06:45.319011 26850 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:06:45.319787 26865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:45.319974 26850 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:06:45.320077 26850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "db8afd05497843978227d3794bac6adf"
format_stamp: "Formatted at 2026-09-29 19:06:45 on dist-test-slave-nmsz"
server_key: "cf1f3967bb2058a93434c49e3f2190b1"
server_key_iv: "0421be70e966c83ef2e194a34be67e5d"
server_key_version: "encryptionkey@0"
I20260929 19:06:45.320151 26850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:06:45.335791 26850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:06:45.337502 26850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:06:45.337653 26850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:06:45.341631 26850 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.126:42691
I20260929 19:06:45.341753 26917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.126:42691 every 8 connection(s)
I20260929 19:06:45.342020 26850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:06:45.342983 26918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:06:45.345209 26918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db8afd05497843978227d3794bac6adf: Bootstrap starting.
I20260929 19:06:45.345743 26918 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db8afd05497843978227d3794bac6adf: Neither blocks nor log segments found. Creating new log.
I20260929 19:06:45.345990 26918 log.cc:824] T 00000000000000000000000000000000 P db8afd05497843978227d3794bac6adf: Log is configured to *not* fsync() on all Append() calls
I20260929 19:06:45.346688 26918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db8afd05497843978227d3794bac6adf: No bootstrap required, opened a new log
I20260929 19:06:45.348348 26918 raft_consensus.cc:348] T 00000000000000000000000000000000 P db8afd05497843978227d3794bac6adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db8afd05497843978227d3794bac6adf" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 42691 } }
I20260929 19:06:45.348490 26918 raft_consensus.cc:374] T 00000000000000000000000000000000 P db8afd05497843978227d3794bac6adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:06:45.348516 26918 raft_consensus.cc:729] T 00000000000000000000000000000000 P db8afd05497843978227d3794bac6adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db8afd05497843978227d3794bac6adf, State: Initialized, Role: FOLLOWER
I20260929 19:06:45.348618 26918 consensus_queue.cc:260] T 00000000000000000000000000000000 P db8afd05497843978227d3794bac6adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db8afd05497843978227d3794bac6adf" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 42691 } }
I20260929 19:06:45.348673 26918 raft_consensus.cc:388] T 00000000000000000000000000000000 P db8afd05497843978227d3794bac6adf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:06:45.348695 26918 raft_consensus.cc:482] T 00000000000000000000000000000000 P db8afd05497843978227d3794bac6adf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:06:45.348723 26918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db8afd05497843978227d3794bac6adf [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:06:45.349218 26918 raft_consensus.cc:504] T 00000000000000000000000000000000 P db8afd05497843978227d3794bac6adf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db8afd05497843978227d3794bac6adf" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 42691 } }
I20260929 19:06:45.349305 26918 leader_election.cc:302] T 00000000000000000000000000000000 P db8afd05497843978227d3794bac6adf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db8afd05497843978227d3794bac6adf; no voters:
I20260929 19:06:45.349427 26918 leader_election.cc:288] T 00000000000000000000000000000000 P db8afd05497843978227d3794bac6adf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:06:45.349481 26921 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db8afd05497843978227d3794bac6adf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:06:45.349658 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 26850
I20260929 19:06:45.349684 26921 raft_consensus.cc:686] T 00000000000000000000000000000000 P db8afd05497843978227d3794bac6adf [term 1 LEADER]: Becoming Leader. State: Replica: db8afd05497843978227d3794bac6adf, State: Running, Role: LEADER
I20260929 19:06:45.349730 26918 sys_catalog.cc:564] T 00000000000000000000000000000000 P db8afd05497843978227d3794bac6adf [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:06:45.349761 26921 consensus_queue.cc:237] T 00000000000000000000000000000000 P db8afd05497843978227d3794bac6adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db8afd05497843978227d3794bac6adf" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 42691 } }
I20260929 19:06:45.349795 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:06:45.350009 8825 external_mini_cluster.cc:1428] Setting key e535134d910a72831e1eeeb4150bba9b
I20260929 19:06:45.350073 26921 sys_catalog.cc:455] T 00000000000000000000000000000000 P db8afd05497843978227d3794bac6adf [sys.catalog]: SysCatalogTable state changed. Reason: New leader db8afd05497843978227d3794bac6adf. Latest consensus state: current_term: 1 leader_uuid: "db8afd05497843978227d3794bac6adf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db8afd05497843978227d3794bac6adf" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 42691 } } }
I20260929 19:06:45.350145 26921 sys_catalog.cc:458] T 00000000000000000000000000000000 P db8afd05497843978227d3794bac6adf [sys.catalog]: This master's current role is: LEADER
I20260929 19:06:45.350219 26923 sys_catalog.cc:455] T 00000000000000000000000000000000 P db8afd05497843978227d3794bac6adf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db8afd05497843978227d3794bac6adf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db8afd05497843978227d3794bac6adf" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 42691 } } }
I20260929 19:06:45.350274 26923 sys_catalog.cc:458] T 00000000000000000000000000000000 P db8afd05497843978227d3794bac6adf [sys.catalog]: This master's current role is: LEADER
I20260929 19:06:45.350426 26926 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:06:45.351142 26926 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:06:45.352576 26926 catalog_manager.cc:1347] Generated new cluster ID: 3b65c1b9bd0e41429dcfb9e35c3c9a05
I20260929 19:06:45.352632 26926 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:06:45.355883 26926 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:06:45.356485 26926 catalog_manager.cc:1504] Loading token signing keys...
Sep 29 19:06:45 dist-test-slave-nmsz krb5kdc[26837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:06:45.363144 26932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:45.350679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51868 (local address 127.8.158.126:42691)
0929 19:06:45.351072 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:45.351083 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:45.351123 (+ 40us) server_negotiation.cc:407] Connection header received
0929 19:06:45.351258 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:45.351265 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:45.352017 (+ 752us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:45.352163 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:45.352937 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:45.354286 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:45.355490 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:45.356147 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:45.356252 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:45.359176 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:45.359180 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:45.359180 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:06:45.359193 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:45.361318 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:45.361705 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:45.361709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:45.361709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:45.361822 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:45.362038 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:45.362039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:45.362039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:45.362359 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:45.362531 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:45.362803 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 19:06:45.362945 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":85,"threads_started":1}
I20260929 19:06:45.365384 26926 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db8afd05497843978227d3794bac6adf: Generated new TSK 0
I20260929 19:06:45.365545 26926 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.158.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab.
Entry for principal kudu/127.8.158.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab.
I20260929 19:06:45.395262 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.65:0
--local_ip_for_outbound_sockets=127.8.158.65
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:42691
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab
--principal=kudu/127.8.158.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:40233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:06:45.474881 26944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:06:45.475061 26944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:06:45.475085 26944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:06:45.476186 26944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:06:45.476224 26944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:06:45.476249 26944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.65
I20260929 19:06:45.477622 26944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:40233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab
--principal=kudu/127.8.158.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:42691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.65
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:06:45.477839 26944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:06:45.478065 26944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:06:45.483647 26950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:45.483668 26952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:45.483817 26949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:45.484053 26944 server_base.cc:1034] running on GCE node
I20260929 19:06:45.484251 26944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:06:45.484434 26944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:06:45.485596 26944 hybrid_clock.cc:648] HybridClock initialized: now 1790708805485584 us; error 23 us; skew 500 ppm
Sep 29 19:06:45 dist-test-slave-nmsz krb5kdc[26837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708805, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:06:45.488204 26944 init.cc:377] Logged in from keytab as kudu/127.8.158.65@KRBTEST.COM (short username kudu)
I20260929 19:06:45.488570 26944 webserver.cc:460] Webserver started at http://127.8.158.65:42905/ using document root <none> and password file <none>
I20260929 19:06:45.488749 26944 fs_manager.cc:359] Metadata directory not provided
I20260929 19:06:45.488786 26944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:06:45.488919 26944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:06:45.489847 26944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-0/data/instance:
uuid: "61ad01aede7e49b0a02e323bfa034e05"
format_stamp: "Formatted at 2026-09-29 19:06:45 on dist-test-slave-nmsz"
server_key: "f0efa248de351b2e287a6a83aae70d13"
server_key_iv: "eaaa4ddc2bfcca4d738009ee4361df82"
server_key_version: "encryptionkey@0"
I20260929 19:06:45.490134 26944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance:
uuid: "61ad01aede7e49b0a02e323bfa034e05"
format_stamp: "Formatted at 2026-09-29 19:06:45 on dist-test-slave-nmsz"
server_key: "f0efa248de351b2e287a6a83aae70d13"
server_key_iv: "eaaa4ddc2bfcca4d738009ee4361df82"
server_key_version: "encryptionkey@0"
I20260929 19:06:45.491405 26944 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:06:45.492089 26959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:45.492237 26944 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:06:45.492324 26944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-0/wal
uuid: "61ad01aede7e49b0a02e323bfa034e05"
format_stamp: "Formatted at 2026-09-29 19:06:45 on dist-test-slave-nmsz"
server_key: "f0efa248de351b2e287a6a83aae70d13"
server_key_iv: "eaaa4ddc2bfcca4d738009ee4361df82"
server_key_version: "encryptionkey@0"
I20260929 19:06:45.492375 26944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:06:45.510463 26944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:06:45.512280 26944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:06:45.512380 26944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:06:45.512535 26944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:06:45.512831 26944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:06:45.512863 26944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:45.512883 26944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:06:45.512936 26944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:45.518618 26944 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.65:44219
I20260929 19:06:45.518653 27072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.65:44219 every 8 connection(s)
I20260929 19:06:45.518957 26944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
I20260929 19:06:45.519371 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 26944
I20260929 19:06:45.519475 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance
I20260929 19:06:45.519704 8825 external_mini_cluster.cc:1428] Setting key dac58862f41f3104025040a980cd2739
Sep 29 19:06:45 dist-test-slave-nmsz krb5kdc[26837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708805, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.158.66@KRBTEST.COM; defaulting to no policy
I20260929 19:06:45.528894 26932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:45.519738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:34493 (local address 127.8.158.126:42691)
0929 19:06:45.519907 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:45.519909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:45.520028 (+ 119us) server_negotiation.cc:407] Connection header received
0929 19:06:45.520568 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:45.520570 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:45.520582 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:45.520651 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:45.521541 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:45.522177 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:45.523382 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:45.523761 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:45.523833 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:45.525969 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:45.525971 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:45.525971 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:06:45.525980 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:45.527650 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:45.528046 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:45.528048 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:45.528048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:45.528107 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:45.528344 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:45.528345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:45.528346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:45.528452 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:45.528514 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:45.528720 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 19:06:45.528778 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260929 19:06:45.529107 27075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:45.519716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:42691 (local address 127.8.158.65:34493)
0929 19:06:45.519897 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:45.519944 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:45.520466 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:45.520699 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:45.520702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:45.521241 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:06:45.521414 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:45.521419 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:45.522270 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:45.522271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:45.523301 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:45.523304 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:45.523866 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:45.523867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:45.524046 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:45.524580 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:06:45.524587 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:06:45.525880 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:06:45.527730 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:06:45.527736 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:06:45.527738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:06:45.527939 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:06:45.528181 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:06:45.528182 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:06:45.528182 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:06:45.528276 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:06:45.528512 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:06:45.528516 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:06:45.528623 (+ 107us) client_negotiation.cc:769] Sending connection context
0929 19:06:45.528706 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 19:06:45.528885 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":87,"threads_started":1}
I20260929 19:06:45.529565 27073 heartbeater.cc:347] Connected to a master server at 127.8.158.126:42691
I20260929 19:06:45.529688 27073 heartbeater.cc:464] Registering TS with master...
I20260929 19:06:45.529876 27073 heartbeater.cc:511] Master 127.8.158.126:42691 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.66@KRBTEST.COM" created.
I20260929 19:06:45.530498 26882 ts_manager.cc:194] Registered new tserver with Master: 61ad01aede7e49b0a02e323bfa034e05 (127.8.158.65:44219)
I20260929 19:06:45.531100 26882 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.65@KRBTEST.COM'} at 127.8.158.65:34493
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab.
Entry for principal kudu/127.8.158.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab.
I20260929 19:06:45.544744 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.66:0
--local_ip_for_outbound_sockets=127.8.158.66
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:42691
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab
--principal=kudu/127.8.158.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:40233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:06:45.622925 27078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:06:45.623121 27078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:06:45.623144 27078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:06:45.624266 27078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:06:45.624306 27078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:06:45.624332 27078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.66
I20260929 19:06:45.625671 27078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:40233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab
--principal=kudu/127.8.158.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:42691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.66
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:06:45.625838 27078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:06:45.625996 27078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:06:45.630628 27086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:45.630676 27083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:45.630769 27078 server_base.cc:1034] running on GCE node
W20260929 19:06:45.630642 27084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:45.630985 27078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:06:45.631208 27078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:06:45.632364 27078 hybrid_clock.cc:648] HybridClock initialized: now 1790708805632348 us; error 45 us; skew 500 ppm
Sep 29 19:06:45 dist-test-slave-nmsz krb5kdc[26837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708805, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:06:45.634630 27078 init.cc:377] Logged in from keytab as kudu/127.8.158.66@KRBTEST.COM (short username kudu)
I20260929 19:06:45.635095 27078 webserver.cc:460] Webserver started at http://127.8.158.66:34299/ using document root <none> and password file <none>
I20260929 19:06:45.635303 27078 fs_manager.cc:359] Metadata directory not provided
I20260929 19:06:45.635368 27078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:06:45.635515 27078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:06:45.636472 27078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-1/data/instance:
uuid: "4661b57ca3eb4d6f98329ee0a5b7c569"
format_stamp: "Formatted at 2026-09-29 19:06:45 on dist-test-slave-nmsz"
server_key: "588c30498093b2da8c03f1641ebb96fd"
server_key_iv: "4c4459f8de33abb7633019b45a7096c5"
server_key_version: "encryptionkey@0"
I20260929 19:06:45.636828 27078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance:
uuid: "4661b57ca3eb4d6f98329ee0a5b7c569"
format_stamp: "Formatted at 2026-09-29 19:06:45 on dist-test-slave-nmsz"
server_key: "588c30498093b2da8c03f1641ebb96fd"
server_key_iv: "4c4459f8de33abb7633019b45a7096c5"
server_key_version: "encryptionkey@0"
I20260929 19:06:45.638128 27078 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:06:45.638906 27093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:45.639065 27078 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:45.639168 27078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-1/wal
uuid: "4661b57ca3eb4d6f98329ee0a5b7c569"
format_stamp: "Formatted at 2026-09-29 19:06:45 on dist-test-slave-nmsz"
server_key: "588c30498093b2da8c03f1641ebb96fd"
server_key_iv: "4c4459f8de33abb7633019b45a7096c5"
server_key_version: "encryptionkey@0"
I20260929 19:06:45.639249 27078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:06:45.653189 27078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:06:45.654932 27078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:06:45.655086 27078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:06:45.655295 27078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:06:45.655575 27078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:06:45.655628 27078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:45.655660 27078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:06:45.655687 27078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:45.661459 27078 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.66:40013
I20260929 19:06:45.661496 27206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.66:40013 every 8 connection(s)
I20260929 19:06:45.661787 27078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
Sep 29 19:06:45 dist-test-slave-nmsz krb5kdc[26837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708805, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.66@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:06:45.669106 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 27078
I20260929 19:06:45.669190 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance
I20260929 19:06:45.669409 8825 external_mini_cluster.cc:1428] Setting key 72a61a63aab998f0a629db4e3491bcd7
I20260929 19:06:45.670734 26932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:45.662494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:37261 (local address 127.8.158.126:42691)
0929 19:06:45.662546 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:45.662548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:45.662760 (+ 212us) server_negotiation.cc:407] Connection header received
0929 19:06:45.663257 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:45.663258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:45.663270 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:45.663322 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:45.664206 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:45.664697 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:45.665667 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:45.665868 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:45.665956 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:45.668000 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:45.668001 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:45.668001 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:06:45.668009 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:45.669256 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:45.669696 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:45.669697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:45.669698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:45.669758 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:45.670040 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:45.670041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:45.670041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:45.670183 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:45.670272 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:45.670569 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 19:06:45.670646 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 19:06:45.670907 27209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:45.662483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:42691 (local address 127.8.158.66:37261)
0929 19:06:45.662645 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:45.662676 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:45.663153 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:45.663383 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:45.663387 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:45.663924 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:06:45.664104 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:45.664108 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:45.664789 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:45.664791 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:45.665571 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:45.665574 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:45.665965 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:45.665966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:45.666096 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:45.666641 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:06:45.666648 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:06:45.667903 (+ 1255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:06:45.669323 (+ 1420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:06:45.669330 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:06:45.669332 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:06:45.669609 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:06:45.669824 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:06:45.669825 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:06:45.669826 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:06:45.669924 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:06:45.670264 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:06:45.670271 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:06:45.670430 (+ 159us) client_negotiation.cc:769] Sending connection context
0929 19:06:45.670532 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 19:06:45.670690 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":76,"threads_started":1}
I20260929 19:06:45.671460 27207 heartbeater.cc:347] Connected to a master server at 127.8.158.126:42691
I20260929 19:06:45.671554 27207 heartbeater.cc:464] Registering TS with master...
I20260929 19:06:45.671723 27207 heartbeater.cc:511] Master 127.8.158.126:42691 requested a full tablet report, sending...
I20260929 19:06:45.672096 26882 ts_manager.cc:194] Registered new tserver with Master: 4661b57ca3eb4d6f98329ee0a5b7c569 (127.8.158.66:40013)
I20260929 19:06:45.672468 26882 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.66@KRBTEST.COM'} at 127.8.158.66:37261
WARNING: no policy specified for kudu/127.8.158.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab.
Entry for principal kudu/127.8.158.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab.
I20260929 19:06:45.692538 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.67:0
--local_ip_for_outbound_sockets=127.8.158.67
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:42691
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab
--principal=kudu/127.8.158.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:40233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:06:45.769603 27212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:06:45.769820 27212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:06:45.769857 27212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:06:45.770987 27212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:06:45.771061 27212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:06:45.771107 27212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.67
I20260929 19:06:45.772497 27212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:40233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab
--principal=kudu/127.8.158.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:42691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.67
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:06:45.772740 27212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:06:45.772953 27212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:06:45.777662 27218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:45.777668 27220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:45.777668 27217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:45.777884 27212 server_base.cc:1034] running on GCE node
I20260929 19:06:45.778085 27212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:06:45.778347 27212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:06:45.779529 27212 hybrid_clock.cc:648] HybridClock initialized: now 1790708805779518 us; error 25 us; skew 500 ppm
Sep 29 19:06:45 dist-test-slave-nmsz krb5kdc[26837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708805, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:06:45.781914 27212 init.cc:377] Logged in from keytab as kudu/127.8.158.67@KRBTEST.COM (short username kudu)
I20260929 19:06:45.782390 27212 webserver.cc:460] Webserver started at http://127.8.158.67:40345/ using document root <none> and password file <none>
I20260929 19:06:45.782604 27212 fs_manager.cc:359] Metadata directory not provided
I20260929 19:06:45.782670 27212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:06:45.782815 27212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:06:45.783849 27212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-2/data/instance:
uuid: "abae53c972bb4662a8d7866cda26fd93"
format_stamp: "Formatted at 2026-09-29 19:06:45 on dist-test-slave-nmsz"
server_key: "71f90feac2d5bc5427f2f8b41de70013"
server_key_iv: "469f3d8ba9215dd4963fea3f76efe075"
server_key_version: "encryptionkey@0"
I20260929 19:06:45.784251 27212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance:
uuid: "abae53c972bb4662a8d7866cda26fd93"
format_stamp: "Formatted at 2026-09-29 19:06:45 on dist-test-slave-nmsz"
server_key: "71f90feac2d5bc5427f2f8b41de70013"
server_key_iv: "469f3d8ba9215dd4963fea3f76efe075"
server_key_version: "encryptionkey@0"
I20260929 19:06:45.785578 27212 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:06:45.786278 27227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:45.786484 27212 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:45.786580 27212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-2/wal
uuid: "abae53c972bb4662a8d7866cda26fd93"
format_stamp: "Formatted at 2026-09-29 19:06:45 on dist-test-slave-nmsz"
server_key: "71f90feac2d5bc5427f2f8b41de70013"
server_key_iv: "469f3d8ba9215dd4963fea3f76efe075"
server_key_version: "encryptionkey@0"
I20260929 19:06:45.786665 27212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:06:45.794719 27212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:06:45.796442 27212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:06:45.796593 27212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:06:45.796797 27212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:06:45.797070 27212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:06:45.797122 27212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:45.797160 27212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:06:45.797188 27212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:45.803052 27212 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.67:43285
I20260929 19:06:45.803102 27340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.67:43285 every 8 connection(s)
I20260929 19:06:45.803385 27212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
I20260929 19:06:45.806725 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 27212
I20260929 19:06:45.806836 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance
I20260929 19:06:45.807003 8825 external_mini_cluster.cc:1428] Setting key 5bd325c0e8ff967e0dd8d29e37cd2a39
Sep 29 19:06:45 dist-test-slave-nmsz krb5kdc[26837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708805, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.67@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:06:45.812021 26932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:45.804126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:42409 (local address 127.8.158.126:42691)
0929 19:06:45.804195 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:45.804197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:45.804374 (+ 177us) server_negotiation.cc:407] Connection header received
0929 19:06:45.804804 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:45.804805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:45.804814 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:45.804877 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:45.805723 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:45.806200 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:45.807335 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:45.807654 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:45.807736 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:45.809702 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:45.809703 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:45.809703 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:06:45.809710 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:45.810898 (+ 1188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:45.811254 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:45.811255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:45.811255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:45.811302 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:45.811561 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:45.811561 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:45.811562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:45.811652 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:45.811702 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:45.811888 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 19:06:45.811944 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 19:06:45.812170 27343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:45.804083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:42691 (local address 127.8.158.67:42409)
0929 19:06:45.804260 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:45.804293 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:45.804718 (+ 425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:45.804953 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:45.804956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:45.805485 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:06:45.805651 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:45.805655 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:45.806276 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:45.806277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:45.807220 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:45.807222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:45.807763 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:45.807764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:45.807896 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:45.808292 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:06:45.808299 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:06:45.809624 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:06:45.810971 (+ 1347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:06:45.810976 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:06:45.810978 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:06:45.811178 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:06:45.811392 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:06:45.811393 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:06:45.811394 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:06:45.811494 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:06:45.811693 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:06:45.811696 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:06:45.811787 (+ 91us) client_negotiation.cc:769] Sending connection context
0929 19:06:45.811858 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 19:06:45.811966 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":97,"threads_started":1}
I20260929 19:06:45.812598 27341 heartbeater.cc:347] Connected to a master server at 127.8.158.126:42691
I20260929 19:06:45.812695 27341 heartbeater.cc:464] Registering TS with master...
I20260929 19:06:45.812850 27341 heartbeater.cc:511] Master 127.8.158.126:42691 requested a full tablet report, sending...
I20260929 19:06:45.813304 26882 ts_manager.cc:194] Registered new tserver with Master: abae53c972bb4662a8d7866cda26fd93 (127.8.158.67:43285)
I20260929 19:06:45.813700 26882 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.67@KRBTEST.COM'} at 127.8.158.67:42409
I20260929 19:06:45.818751 8825 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:06:45.825868 26932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:45.820093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51884 (local address 127.8.158.126:42691)
0929 19:06:45.820128 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:45.820130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:45.820255 (+ 125us) server_negotiation.cc:407] Connection header received
0929 19:06:45.820277 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:45.820278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:45.820286 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:45.820336 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:45.820827 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:45.821264 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:45.821945 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:45.822121 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:45.822168 (+ 47us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:45.823499 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:45.823500 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:45.823500 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:06:45.823506 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:45.824828 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:45.825201 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:45.825201 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:45.825202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:45.825249 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:45.825428 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:45.825428 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:45.825429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:45.825536 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:45.825612 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:45.825714 (+ 102us) server_negotiation.cc:299] Negotiation successful
0929 19:06:45.825756 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 19:06:45.827527 26882 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51884:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:06:45.828935 26882 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:06:45.834527 27353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:45.831006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42742 (local address 127.8.158.67:43285)
0929 19:06:45.831153 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:45.831157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:45.831168 (+ 11us) server_negotiation.cc:407] Connection header received
0929 19:06:45.831205 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:45.831207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:45.831222 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:45.831303 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:06:45.831540 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:45.832343 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:45.833442 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:45.834132 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:45.834224 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:45.834285 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:45.834421 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 19:06:45.834456 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":93,"threads_started":1}
I20260929 19:06:45.834530 27351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:45.830858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.67:43285 (local address 127.0.0.1:42742)
0929 19:06:45.831027 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:45.831065 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:45.831126 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:45.831390 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:45.831393 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:45.831395 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:06:45.831463 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:45.831465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:45.832444 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:45.832445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:45.833355 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:45.833357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:45.834225 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:45.834226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:45.834302 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:45.834334 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 19:06:45.834407 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 19:06:45.834454 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":55,"threads_started":1}
I20260929 19:06:45.835207 27350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:45.830863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:44219 (local address 127.0.0.1:33866)
0929 19:06:45.831223 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:45.831230 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:45.831257 (+ 27us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:45.831837 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:45.831838 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:45.831839 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:06:45.831867 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:45.831869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:45.833074 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:45.833075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:45.834069 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:45.834072 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:45.834967 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:45.834968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:45.835046 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:45.835049 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:06:45.835105 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 19:06:45.835143 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":28,"thread_start_us":54,"threads_started":1}
I20260929 19:06:45.835225 27352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:45.831010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33866 (local address 127.8.158.65:44219)
0929 19:06:45.831624 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:45.831628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:45.831638 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:06:45.831673 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:45.831676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:45.831691 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:45.831762 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:06:45.831942 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:45.832977 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:45.834151 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:45.834871 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:45.834965 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:45.835016 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:45.835118 (+ 102us) server_negotiation.cc:299] Negotiation successful
0929 19:06:45.835154 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":84,"threads_started":1}
I20260929 19:06:45.835314 27349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:45.830868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.66:40013 (local address 127.0.0.1:48856)
0929 19:06:45.831363 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:45.831373 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:45.831427 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:45.831717 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:45.831718 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:45.831720 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:06:45.831763 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:45.831765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:45.833045 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:45.833046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:45.834065 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:45.834068 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:45.835083 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:45.835084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:45.835155 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:45.835159 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:06:45.835209 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 19:06:45.835249 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":1,"thread_start_us":66,"threads_started":1}
I20260929 19:06:45.835494 27354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:45.831192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48856 (local address 127.8.158.66:40013)
0929 19:06:45.831367 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:45.831371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:45.831416 (+ 45us) server_negotiation.cc:407] Connection header received
0929 19:06:45.831494 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:45.831515 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:45.831531 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:45.831611 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:06:45.831843 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:45.832935 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:45.834167 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:45.834924 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:45.834996 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:45.835049 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:45.835375 (+ 326us) server_negotiation.cc:299] Negotiation successful
0929 19:06:45.835417 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":83,"threads_started":1}
I20260929 19:06:45.835474 27275 tablet_service.cc:1467] Processing CreateTablet for tablet c8d4bd7d07aa4bbcae941e930feff460 (DEFAULT_TABLE table=test-table [id=3592305cb2e34d308cfd2f76e3a43fe4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:06:45.835812 27275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8d4bd7d07aa4bbcae941e930feff460. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:06:45.836119 27007 tablet_service.cc:1467] Processing CreateTablet for tablet c8d4bd7d07aa4bbcae941e930feff460 (DEFAULT_TABLE table=test-table [id=3592305cb2e34d308cfd2f76e3a43fe4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:06:45.836234 27141 tablet_service.cc:1467] Processing CreateTablet for tablet c8d4bd7d07aa4bbcae941e930feff460 (DEFAULT_TABLE table=test-table [id=3592305cb2e34d308cfd2f76e3a43fe4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:06:45.836437 27007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8d4bd7d07aa4bbcae941e930feff460. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:06:45.836552 27141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8d4bd7d07aa4bbcae941e930feff460. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:06:45.838989 27355 tablet_bootstrap.cc:492] T c8d4bd7d07aa4bbcae941e930feff460 P 61ad01aede7e49b0a02e323bfa034e05: Bootstrap starting.
I20260929 19:06:45.839143 27356 tablet_bootstrap.cc:492] T c8d4bd7d07aa4bbcae941e930feff460 P abae53c972bb4662a8d7866cda26fd93: Bootstrap starting.
I20260929 19:06:45.839646 27357 tablet_bootstrap.cc:492] T c8d4bd7d07aa4bbcae941e930feff460 P 4661b57ca3eb4d6f98329ee0a5b7c569: Bootstrap starting.
I20260929 19:06:45.839728 27355 tablet_bootstrap.cc:654] T c8d4bd7d07aa4bbcae941e930feff460 P 61ad01aede7e49b0a02e323bfa034e05: Neither blocks nor log segments found. Creating new log.
I20260929 19:06:45.839744 27356 tablet_bootstrap.cc:654] T c8d4bd7d07aa4bbcae941e930feff460 P abae53c972bb4662a8d7866cda26fd93: Neither blocks nor log segments found. Creating new log.
I20260929 19:06:45.840214 27357 tablet_bootstrap.cc:654] T c8d4bd7d07aa4bbcae941e930feff460 P 4661b57ca3eb4d6f98329ee0a5b7c569: Neither blocks nor log segments found. Creating new log.
I20260929 19:06:45.840306 27356 log.cc:824] T c8d4bd7d07aa4bbcae941e930feff460 P abae53c972bb4662a8d7866cda26fd93: Log is configured to *not* fsync() on all Append() calls
I20260929 19:06:45.840404 27355 log.cc:824] T c8d4bd7d07aa4bbcae941e930feff460 P 61ad01aede7e49b0a02e323bfa034e05: Log is configured to *not* fsync() on all Append() calls
I20260929 19:06:45.840758 27357 log.cc:824] T c8d4bd7d07aa4bbcae941e930feff460 P 4661b57ca3eb4d6f98329ee0a5b7c569: Log is configured to *not* fsync() on all Append() calls
I20260929 19:06:45.841045 27355 tablet_bootstrap.cc:492] T c8d4bd7d07aa4bbcae941e930feff460 P 61ad01aede7e49b0a02e323bfa034e05: No bootstrap required, opened a new log
I20260929 19:06:45.841066 27356 tablet_bootstrap.cc:492] T c8d4bd7d07aa4bbcae941e930feff460 P abae53c972bb4662a8d7866cda26fd93: No bootstrap required, opened a new log
I20260929 19:06:45.841115 27355 ts_tablet_manager.cc:1397] T c8d4bd7d07aa4bbcae941e930feff460 P 61ad01aede7e49b0a02e323bfa034e05: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:06:45.841125 27356 ts_tablet_manager.cc:1397] T c8d4bd7d07aa4bbcae941e930feff460 P abae53c972bb4662a8d7866cda26fd93: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:06:45.841569 27357 tablet_bootstrap.cc:492] T c8d4bd7d07aa4bbcae941e930feff460 P 4661b57ca3eb4d6f98329ee0a5b7c569: No bootstrap required, opened a new log
I20260929 19:06:45.841686 27357 ts_tablet_manager.cc:1397] T c8d4bd7d07aa4bbcae941e930feff460 P 4661b57ca3eb4d6f98329ee0a5b7c569: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:06:45.842988 27355 raft_consensus.cc:348] T c8d4bd7d07aa4bbcae941e930feff460 P 61ad01aede7e49b0a02e323bfa034e05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abae53c972bb4662a8d7866cda26fd93" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 43285 } } peers { permanent_uuid: "61ad01aede7e49b0a02e323bfa034e05" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 44219 } } peers { permanent_uuid: "4661b57ca3eb4d6f98329ee0a5b7c569" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 40013 } }
I20260929 19:06:45.843160 27355 raft_consensus.cc:374] T c8d4bd7d07aa4bbcae941e930feff460 P 61ad01aede7e49b0a02e323bfa034e05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:06:45.843104 27356 raft_consensus.cc:348] T c8d4bd7d07aa4bbcae941e930feff460 P abae53c972bb4662a8d7866cda26fd93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abae53c972bb4662a8d7866cda26fd93" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 43285 } } peers { permanent_uuid: "61ad01aede7e49b0a02e323bfa034e05" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 44219 } } peers { permanent_uuid: "4661b57ca3eb4d6f98329ee0a5b7c569" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 40013 } }
I20260929 19:06:45.843219 27355 raft_consensus.cc:729] T c8d4bd7d07aa4bbcae941e930feff460 P 61ad01aede7e49b0a02e323bfa034e05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61ad01aede7e49b0a02e323bfa034e05, State: Initialized, Role: FOLLOWER
I20260929 19:06:45.843271 27356 raft_consensus.cc:374] T c8d4bd7d07aa4bbcae941e930feff460 P abae53c972bb4662a8d7866cda26fd93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:06:45.843313 27356 raft_consensus.cc:729] T c8d4bd7d07aa4bbcae941e930feff460 P abae53c972bb4662a8d7866cda26fd93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abae53c972bb4662a8d7866cda26fd93, State: Initialized, Role: FOLLOWER
I20260929 19:06:45.843353 27355 consensus_queue.cc:260] T c8d4bd7d07aa4bbcae941e930feff460 P 61ad01aede7e49b0a02e323bfa034e05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abae53c972bb4662a8d7866cda26fd93" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 43285 } } peers { permanent_uuid: "61ad01aede7e49b0a02e323bfa034e05" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 44219 } } peers { permanent_uuid: "4661b57ca3eb4d6f98329ee0a5b7c569" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 40013 } }
I20260929 19:06:45.843420 27356 consensus_queue.cc:260] T c8d4bd7d07aa4bbcae941e930feff460 P abae53c972bb4662a8d7866cda26fd93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abae53c972bb4662a8d7866cda26fd93" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 43285 } } peers { permanent_uuid: "61ad01aede7e49b0a02e323bfa034e05" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 44219 } } peers { permanent_uuid: "4661b57ca3eb4d6f98329ee0a5b7c569" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 40013 } }
I20260929 19:06:45.843657 27355 ts_tablet_manager.cc:1428] T c8d4bd7d07aa4bbcae941e930feff460 P 61ad01aede7e49b0a02e323bfa034e05: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 19:06:45.843677 27073 heartbeater.cc:503] Master 127.8.158.126:42691 was elected leader, sending a full tablet report...
I20260929 19:06:45.843681 27356 ts_tablet_manager.cc:1428] T c8d4bd7d07aa4bbcae941e930feff460 P abae53c972bb4662a8d7866cda26fd93: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 19:06:45.843806 27341 heartbeater.cc:503] Master 127.8.158.126:42691 was elected leader, sending a full tablet report...
I20260929 19:06:45.844010 27357 raft_consensus.cc:348] T c8d4bd7d07aa4bbcae941e930feff460 P 4661b57ca3eb4d6f98329ee0a5b7c569 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abae53c972bb4662a8d7866cda26fd93" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 43285 } } peers { permanent_uuid: "61ad01aede7e49b0a02e323bfa034e05" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 44219 } } peers { permanent_uuid: "4661b57ca3eb4d6f98329ee0a5b7c569" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 40013 } }
I20260929 19:06:45.844178 27357 raft_consensus.cc:374] T c8d4bd7d07aa4bbcae941e930feff460 P 4661b57ca3eb4d6f98329ee0a5b7c569 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:06:45.844214 27357 raft_consensus.cc:729] T c8d4bd7d07aa4bbcae941e930feff460 P 4661b57ca3eb4d6f98329ee0a5b7c569 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4661b57ca3eb4d6f98329ee0a5b7c569, State: Initialized, Role: FOLLOWER
I20260929 19:06:45.844319 27357 consensus_queue.cc:260] T c8d4bd7d07aa4bbcae941e930feff460 P 4661b57ca3eb4d6f98329ee0a5b7c569 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abae53c972bb4662a8d7866cda26fd93" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 43285 } } peers { permanent_uuid: "61ad01aede7e49b0a02e323bfa034e05" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 44219 } } peers { permanent_uuid: "4661b57ca3eb4d6f98329ee0a5b7c569" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 40013 } }
I20260929 19:06:45.844571 27357 ts_tablet_manager.cc:1428] T c8d4bd7d07aa4bbcae941e930feff460 P 4661b57ca3eb4d6f98329ee0a5b7c569: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 19:06:45.844641 27207 heartbeater.cc:503] Master 127.8.158.126:42691 was elected leader, sending a full tablet report...
W20260929 19:06:45.912333 27208 tablet.cc:2369] T c8d4bd7d07aa4bbcae941e930feff460 P 4661b57ca3eb4d6f98329ee0a5b7c569: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 19:06:46.019671 27074 tablet.cc:2369] T c8d4bd7d07aa4bbcae941e930feff460 P 61ad01aede7e49b0a02e323bfa034e05: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 19:06:46.053926 27342 tablet.cc:2369] T c8d4bd7d07aa4bbcae941e930feff460 P abae53c972bb4662a8d7866cda26fd93: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:06:46.253636 27363 raft_consensus.cc:482] T c8d4bd7d07aa4bbcae941e930feff460 P 4661b57ca3eb4d6f98329ee0a5b7c569 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:06:46.253758 27363 raft_consensus.cc:504] T c8d4bd7d07aa4bbcae941e930feff460 P 4661b57ca3eb4d6f98329ee0a5b7c569 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abae53c972bb4662a8d7866cda26fd93" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 43285 } } peers { permanent_uuid: "61ad01aede7e49b0a02e323bfa034e05" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 44219 } } peers { permanent_uuid: "4661b57ca3eb4d6f98329ee0a5b7c569" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 40013 } }
I20260929 19:06:46.254091 27363 leader_election.cc:288] T c8d4bd7d07aa4bbcae941e930feff460 P 4661b57ca3eb4d6f98329ee0a5b7c569 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abae53c972bb4662a8d7866cda26fd93 (127.8.158.67:43285), 61ad01aede7e49b0a02e323bfa034e05 (127.8.158.65:44219)
I20260929 19:06:46.257030 27352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:46.254347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:52601 (local address 127.8.158.65:44219)
0929 19:06:46.254395 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:46.254397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:46.254501 (+ 104us) server_negotiation.cc:407] Connection header received
0929 19:06:46.254585 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:46.254590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:46.254599 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:46.254656 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:06:46.254824 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:46.255305 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:46.256313 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:46.256671 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:46.256758 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:46.256766 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:46.256921 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 19:06:46.256960 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 19:06:46.257025 27353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:46.254396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:36245 (local address 127.8.158.67:43285)
0929 19:06:46.254444 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:46.254446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:46.254514 (+ 68us) server_negotiation.cc:407] Connection header received
0929 19:06:46.254593 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:46.254595 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:46.254604 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:46.254681 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:06:46.254838 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:46.255314 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:46.256305 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:46.256677 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:46.256763 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:46.256771 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:46.256923 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 19:06:46.256958 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 19:06:46.257056 27364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:46.254254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:44219 (local address 127.8.158.66:52601)
0929 19:06:46.254411 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:46.254422 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:46.254509 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:46.254674 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:46.254675 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:46.254677 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:06:46.254752 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:46.254754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:46.255390 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:46.255391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:46.256213 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:46.256215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:46.256755 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:46.256756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:46.256836 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:46.256839 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:06:46.256903 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 19:06:46.256947 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":57,"thread_start_us":82,"threads_started":1}
I20260929 19:06:46.257059 27365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:46.254271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.67:43285 (local address 127.8.158.66:36245)
0929 19:06:46.254431 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:46.254441 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:46.254509 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:46.254686 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:46.254687 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:46.254688 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:06:46.254766 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:46.254768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:46.255402 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:46.255403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:46.256210 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:46.256212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:46.256763 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:46.256764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:46.256842 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:46.256845 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:06:46.256905 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 19:06:46.256949 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":1,"thread_start_us":94,"threads_started":1}
I20260929 19:06:46.257397 27027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8d4bd7d07aa4bbcae941e930feff460" candidate_uuid: "4661b57ca3eb4d6f98329ee0a5b7c569" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61ad01aede7e49b0a02e323bfa034e05" is_pre_election: true
I20260929 19:06:46.257560 27027 raft_consensus.cc:2454] T c8d4bd7d07aa4bbcae941e930feff460 P 61ad01aede7e49b0a02e323bfa034e05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4661b57ca3eb4d6f98329ee0a5b7c569 in term 0.
I20260929 19:06:46.257570 27295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8d4bd7d07aa4bbcae941e930feff460" candidate_uuid: "4661b57ca3eb4d6f98329ee0a5b7c569" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abae53c972bb4662a8d7866cda26fd93" is_pre_election: true
I20260929 19:06:46.257685 27295 raft_consensus.cc:2454] T c8d4bd7d07aa4bbcae941e930feff460 P abae53c972bb4662a8d7866cda26fd93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4661b57ca3eb4d6f98329ee0a5b7c569 in term 0.
I20260929 19:06:46.257825 27096 leader_election.cc:302] T c8d4bd7d07aa4bbcae941e930feff460 P 4661b57ca3eb4d6f98329ee0a5b7c569 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4661b57ca3eb4d6f98329ee0a5b7c569, 61ad01aede7e49b0a02e323bfa034e05; no voters:
I20260929 19:06:46.257936 27363 raft_consensus.cc:2781] T c8d4bd7d07aa4bbcae941e930feff460 P 4661b57ca3eb4d6f98329ee0a5b7c569 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:06:46.257988 27363 raft_consensus.cc:482] T c8d4bd7d07aa4bbcae941e930feff460 P 4661b57ca3eb4d6f98329ee0a5b7c569 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:06:46.258018 27363 raft_consensus.cc:3037] T c8d4bd7d07aa4bbcae941e930feff460 P 4661b57ca3eb4d6f98329ee0a5b7c569 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:06:46.258754 27363 raft_consensus.cc:504] T c8d4bd7d07aa4bbcae941e930feff460 P 4661b57ca3eb4d6f98329ee0a5b7c569 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abae53c972bb4662a8d7866cda26fd93" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 43285 } } peers { permanent_uuid: "61ad01aede7e49b0a02e323bfa034e05" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 44219 } } peers { permanent_uuid: "4661b57ca3eb4d6f98329ee0a5b7c569" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 40013 } }
I20260929 19:06:46.258901 27363 leader_election.cc:288] T c8d4bd7d07aa4bbcae941e930feff460 P 4661b57ca3eb4d6f98329ee0a5b7c569 [CANDIDATE]: Term 1 election: Requested vote from peers abae53c972bb4662a8d7866cda26fd93 (127.8.158.67:43285), 61ad01aede7e49b0a02e323bfa034e05 (127.8.158.65:44219)
I20260929 19:06:46.259009 27295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8d4bd7d07aa4bbcae941e930feff460" candidate_uuid: "4661b57ca3eb4d6f98329ee0a5b7c569" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abae53c972bb4662a8d7866cda26fd93"
I20260929 19:06:46.259094 27295 raft_consensus.cc:3037] T c8d4bd7d07aa4bbcae941e930feff460 P abae53c972bb4662a8d7866cda26fd93 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:06:46.259063 27027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8d4bd7d07aa4bbcae941e930feff460" candidate_uuid: "4661b57ca3eb4d6f98329ee0a5b7c569" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61ad01aede7e49b0a02e323bfa034e05"
I20260929 19:06:46.259176 27027 raft_consensus.cc:3037] T c8d4bd7d07aa4bbcae941e930feff460 P 61ad01aede7e49b0a02e323bfa034e05 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:06:46.259764 27295 raft_consensus.cc:2454] T c8d4bd7d07aa4bbcae941e930feff460 P abae53c972bb4662a8d7866cda26fd93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4661b57ca3eb4d6f98329ee0a5b7c569 in term 1.
I20260929 19:06:46.259882 27027 raft_consensus.cc:2454] T c8d4bd7d07aa4bbcae941e930feff460 P 61ad01aede7e49b0a02e323bfa034e05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4661b57ca3eb4d6f98329ee0a5b7c569 in term 1.
I20260929 19:06:46.259934 27094 leader_election.cc:302] T c8d4bd7d07aa4bbcae941e930feff460 P 4661b57ca3eb4d6f98329ee0a5b7c569 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4661b57ca3eb4d6f98329ee0a5b7c569, abae53c972bb4662a8d7866cda26fd93; no voters:
I20260929 19:06:46.260020 27363 raft_consensus.cc:2781] T c8d4bd7d07aa4bbcae941e930feff460 P 4661b57ca3eb4d6f98329ee0a5b7c569 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:06:46.260169 27363 raft_consensus.cc:686] T c8d4bd7d07aa4bbcae941e930feff460 P 4661b57ca3eb4d6f98329ee0a5b7c569 [term 1 LEADER]: Becoming Leader. State: Replica: 4661b57ca3eb4d6f98329ee0a5b7c569, State: Running, Role: LEADER
I20260929 19:06:46.260273 27363 consensus_queue.cc:237] T c8d4bd7d07aa4bbcae941e930feff460 P 4661b57ca3eb4d6f98329ee0a5b7c569 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abae53c972bb4662a8d7866cda26fd93" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 43285 } } peers { permanent_uuid: "61ad01aede7e49b0a02e323bfa034e05" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 44219 } } peers { permanent_uuid: "4661b57ca3eb4d6f98329ee0a5b7c569" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 40013 } }
I20260929 19:06:46.261021 26882 catalog_manager.cc:5495] T c8d4bd7d07aa4bbcae941e930feff460 P 4661b57ca3eb4d6f98329ee0a5b7c569 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4661b57ca3eb4d6f98329ee0a5b7c569 (127.8.158.66). New cstate: current_term: 1 leader_uuid: "4661b57ca3eb4d6f98329ee0a5b7c569" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abae53c972bb4662a8d7866cda26fd93" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61ad01aede7e49b0a02e323bfa034e05" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 44219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4661b57ca3eb4d6f98329ee0a5b7c569" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 40013 } health_report { overall_health: HEALTHY } } }
I20260929 19:06:46.273542 27354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:46.271498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48870 (local address 127.8.158.66:40013)
0929 19:06:46.271550 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:46.271552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:46.271560 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:06:46.271587 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:46.271588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:46.271611 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:46.271672 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:06:46.271830 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:46.272247 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:46.272919 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:46.273106 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:46.273158 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:46.273300 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:06:46.273375 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:06:46.273451 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:46.273490 (+ 39us) server_negotiation.cc:299] Negotiation successful
0929 19:06:46.273498 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 19:06:46.275131 27027 raft_consensus.cc:1261] T c8d4bd7d07aa4bbcae941e930feff460 P 61ad01aede7e49b0a02e323bfa034e05 [term 1 FOLLOWER]: Refusing update from remote peer 4661b57ca3eb4d6f98329ee0a5b7c569: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:06:46.275184 27295 raft_consensus.cc:1261] T c8d4bd7d07aa4bbcae941e930feff460 P abae53c972bb4662a8d7866cda26fd93 [term 1 FOLLOWER]: Refusing update from remote peer 4661b57ca3eb4d6f98329ee0a5b7c569: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:06:46.275501 27366 consensus_queue.cc:1035] T c8d4bd7d07aa4bbcae941e930feff460 P 4661b57ca3eb4d6f98329ee0a5b7c569 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61ad01aede7e49b0a02e323bfa034e05" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 44219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:06:46.275621 27363 consensus_queue.cc:1035] T c8d4bd7d07aa4bbcae941e930feff460 P 4661b57ca3eb4d6f98329ee0a5b7c569 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abae53c972bb4662a8d7866cda26fd93" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:06:46.278487 27371 mvcc.cc:205] Tried to move back new op lower bound from 7334743270500622336 to 7334743270442483712. Current Snapshot: MvccSnapshot[applied={T|T < 7334743270500622336}]
I20260929 19:06:46.278784 27370 mvcc.cc:205] Tried to move back new op lower bound from 7334743270500622336 to 7334743270442483712. Current Snapshot: MvccSnapshot[applied={T|T < 7334743270500622336 or (T in {7334743270500622336})}]
I20260929 19:06:46.278920 27369 mvcc.cc:205] Tried to move back new op lower bound from 7334743270500622336 to 7334743270442483712. Current Snapshot: MvccSnapshot[applied={T|T < 7334743270500622336 or (T in {7334743270500622336})}]
I20260929 19:06:46.279862 26882 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:06:46.279951 26882 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:06:46.280536 26882 catalog_manager.cc:5782] T 00000000000000000000000000000000 P db8afd05497843978227d3794bac6adf: Sending DeleteTablet for 3 replicas of tablet c8d4bd7d07aa4bbcae941e930feff460
I20260929 19:06:46.280894 27141 tablet_service.cc:1514] Processing DeleteTablet for tablet c8d4bd7d07aa4bbcae941e930feff460 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:06:46 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:48856
I20260929 19:06:46.280943 27275 tablet_service.cc:1514] Processing DeleteTablet for tablet c8d4bd7d07aa4bbcae941e930feff460 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:06:46 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:42742
I20260929 19:06:46.281114 27380 tablet_replica.cc:330] T c8d4bd7d07aa4bbcae941e930feff460 P 4661b57ca3eb4d6f98329ee0a5b7c569: stopping tablet replica
I20260929 19:06:46.281124 27007 tablet_service.cc:1514] Processing DeleteTablet for tablet c8d4bd7d07aa4bbcae941e930feff460 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:06:46 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:33866
I20260929 19:06:46.281217 27380 raft_consensus.cc:2229] T c8d4bd7d07aa4bbcae941e930feff460 P 4661b57ca3eb4d6f98329ee0a5b7c569 [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:06:46.281302 27381 tablet_replica.cc:330] T c8d4bd7d07aa4bbcae941e930feff460 P 61ad01aede7e49b0a02e323bfa034e05: stopping tablet replica
I20260929 19:06:46.281392 27380 raft_consensus.cc:2258] T c8d4bd7d07aa4bbcae941e930feff460 P 4661b57ca3eb4d6f98329ee0a5b7c569 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:06:46.281427 27381 raft_consensus.cc:2229] T c8d4bd7d07aa4bbcae941e930feff460 P 61ad01aede7e49b0a02e323bfa034e05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:06:46.281529 27381 raft_consensus.cc:2258] T c8d4bd7d07aa4bbcae941e930feff460 P 61ad01aede7e49b0a02e323bfa034e05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:06:46.281541 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 26944
I20260929 19:06:46.281899 27380 ts_tablet_manager.cc:1905] T c8d4bd7d07aa4bbcae941e930feff460 P 4661b57ca3eb4d6f98329ee0a5b7c569: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:06:46.284189 27380 ts_tablet_manager.cc:1918] T c8d4bd7d07aa4bbcae941e930feff460 P 4661b57ca3eb4d6f98329ee0a5b7c569: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:06:46.284265 27380 log.cc:1196] T c8d4bd7d07aa4bbcae941e930feff460 P 4661b57ca3eb4d6f98329ee0a5b7c569: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-1/wal/wals/c8d4bd7d07aa4bbcae941e930feff460
I20260929 19:06:46.284550 27380 ts_tablet_manager.cc:1939] T c8d4bd7d07aa4bbcae941e930feff460 P 4661b57ca3eb4d6f98329ee0a5b7c569: Deleting consensus metadata
I20260929 19:06:46.285156 26869 catalog_manager.cc:4841] TS 4661b57ca3eb4d6f98329ee0a5b7c569 (127.8.158.66:40013): tablet c8d4bd7d07aa4bbcae941e930feff460 (table test-table [id=3592305cb2e34d308cfd2f76e3a43fe4]) successfully deleted
W20260929 19:06:46.288489 26868 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.158.65:44219 (error 108)
I20260929 19:06:46.288512 27379 tablet_replica.cc:330] T c8d4bd7d07aa4bbcae941e930feff460 P abae53c972bb4662a8d7866cda26fd93: stopping tablet replica
I20260929 19:06:46.288604 27379 raft_consensus.cc:2229] T c8d4bd7d07aa4bbcae941e930feff460 P abae53c972bb4662a8d7866cda26fd93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:06:46.288692 27379 raft_consensus.cc:2258] T c8d4bd7d07aa4bbcae941e930feff460 P abae53c972bb4662a8d7866cda26fd93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:06:46.288708 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 27078
I20260929 19:06:46.289585 27379 ts_tablet_manager.cc:1905] T c8d4bd7d07aa4bbcae941e930feff460 P abae53c972bb4662a8d7866cda26fd93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:06:46.291028 27350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:46.290891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:44219 (local address 127.0.0.1:33876)
0929 19:06:46.290950 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:46.290990 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.65:44219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 19:06:46.291167 26868 catalog_manager.cc:4584] TS 61ad01aede7e49b0a02e323bfa034e05 (127.8.158.65:44219): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c8d4bd7d07aa4bbcae941e930feff460: Network error: Client connection negotiation failed: client connection to 127.8.158.65:44219: connect: Connection refused (error 111)
I20260929 19:06:46.291388 27379 ts_tablet_manager.cc:1918] T c8d4bd7d07aa4bbcae941e930feff460 P abae53c972bb4662a8d7866cda26fd93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:06:46.291456 27379 log.cc:1196] T c8d4bd7d07aa4bbcae941e930feff460 P abae53c972bb4662a8d7866cda26fd93: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708573736807-8825-0/minicluster-data/ts-2/wal/wals/c8d4bd7d07aa4bbcae941e930feff460
I20260929 19:06:46.291697 27379 ts_tablet_manager.cc:1939] T c8d4bd7d07aa4bbcae941e930feff460 P abae53c972bb4662a8d7866cda26fd93: Deleting consensus metadata
I20260929 19:06:46.292065 26866 catalog_manager.cc:4841] TS abae53c972bb4662a8d7866cda26fd93 (127.8.158.67:43285): tablet c8d4bd7d07aa4bbcae941e930feff460 (table test-table [id=3592305cb2e34d308cfd2f76e3a43fe4]) successfully deleted
I20260929 19:06:46.294368 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 27212
I20260929 19:06:46.298486 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 26850
2026-09-29T19:06:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2234 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T19:06:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:06:46Z Disabled control of system clock
I20260929 19:06:46.324937 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:44925
--webserver_interface=127.8.158.126
--webserver_port=0
--builtin_ntp_servers=127.8.158.84:43631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:44925
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 19:06:46.403753 27386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:06:46.403947 27386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:06:46.403967 27386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:06:46.405027 27386 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:06:46.405061 27386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:06:46.405073 27386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:06:46.405086 27386 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:06:46.405097 27386 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:06:46.406610 27386 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:43631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:44925
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:44925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.158.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:06:46.406800 27386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:06:46.407021 27386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:06:46.411588 27394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:46.411685 27386 server_base.cc:1034] running on GCE node
W20260929 19:06:46.411599 27392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:46.411620 27391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:46.412067 27386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:06:46.412333 27386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:06:46.413506 27386 hybrid_clock.cc:648] HybridClock initialized: now 1790708806413500 us; error 37 us; skew 500 ppm
I20260929 19:06:46.414722 27386 webserver.cc:460] Webserver started at http://127.8.158.126:37025/ using document root <none> and password file <none>
I20260929 19:06:46.414948 27386 fs_manager.cc:359] Metadata directory not provided
I20260929 19:06:46.415016 27386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:06:46.415166 27386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:06:46.416131 27386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "f782409ea8ad452ab278574c3bcc5c45"
format_stamp: "Formatted at 2026-09-29 19:06:46 on dist-test-slave-nmsz"
server_key: "b4ee6118a649aa33306465a29fde1079"
server_key_iv: "8e35852f5ee80b5b5d89869bd17f4e74"
server_key_version: "encryptionkey@0"
I20260929 19:06:46.416484 27386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "f782409ea8ad452ab278574c3bcc5c45"
format_stamp: "Formatted at 2026-09-29 19:06:46 on dist-test-slave-nmsz"
server_key: "b4ee6118a649aa33306465a29fde1079"
server_key_iv: "8e35852f5ee80b5b5d89869bd17f4e74"
server_key_version: "encryptionkey@0"
I20260929 19:06:46.417701 27386 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:06:46.418480 27400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:46.418666 27386 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:06:46.418740 27386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "f782409ea8ad452ab278574c3bcc5c45"
format_stamp: "Formatted at 2026-09-29 19:06:46 on dist-test-slave-nmsz"
server_key: "b4ee6118a649aa33306465a29fde1079"
server_key_iv: "8e35852f5ee80b5b5d89869bd17f4e74"
server_key_version: "encryptionkey@0"
I20260929 19:06:46.418823 27386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:06:46.444965 27386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:06:46.445256 27386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:06:46.445406 27386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:06:46.449309 27386 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.126:44925
I20260929 19:06:46.449383 27452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.126:44925 every 8 connection(s)
I20260929 19:06:46.449671 27386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:06:46.450615 27453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:06:46.452754 27453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f782409ea8ad452ab278574c3bcc5c45: Bootstrap starting.
I20260929 19:06:46.453303 27453 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f782409ea8ad452ab278574c3bcc5c45: Neither blocks nor log segments found. Creating new log.
I20260929 19:06:46.453598 27453 log.cc:824] T 00000000000000000000000000000000 P f782409ea8ad452ab278574c3bcc5c45: Log is configured to *not* fsync() on all Append() calls
I20260929 19:06:46.454224 27453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f782409ea8ad452ab278574c3bcc5c45: No bootstrap required, opened a new log
I20260929 19:06:46.455778 27453 raft_consensus.cc:348] T 00000000000000000000000000000000 P f782409ea8ad452ab278574c3bcc5c45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f782409ea8ad452ab278574c3bcc5c45" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 44925 } }
I20260929 19:06:46.455907 27453 raft_consensus.cc:374] T 00000000000000000000000000000000 P f782409ea8ad452ab278574c3bcc5c45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:06:46.455956 27453 raft_consensus.cc:729] T 00000000000000000000000000000000 P f782409ea8ad452ab278574c3bcc5c45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f782409ea8ad452ab278574c3bcc5c45, State: Initialized, Role: FOLLOWER
I20260929 19:06:46.456101 27453 consensus_queue.cc:260] T 00000000000000000000000000000000 P f782409ea8ad452ab278574c3bcc5c45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f782409ea8ad452ab278574c3bcc5c45" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 44925 } }
I20260929 19:06:46.456162 27453 raft_consensus.cc:388] T 00000000000000000000000000000000 P f782409ea8ad452ab278574c3bcc5c45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:06:46.456207 27453 raft_consensus.cc:482] T 00000000000000000000000000000000 P f782409ea8ad452ab278574c3bcc5c45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:06:46.456257 27453 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f782409ea8ad452ab278574c3bcc5c45 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:06:46.456796 27453 raft_consensus.cc:504] T 00000000000000000000000000000000 P f782409ea8ad452ab278574c3bcc5c45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f782409ea8ad452ab278574c3bcc5c45" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 44925 } }
I20260929 19:06:46.456905 27453 leader_election.cc:302] T 00000000000000000000000000000000 P f782409ea8ad452ab278574c3bcc5c45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f782409ea8ad452ab278574c3bcc5c45; no voters:
I20260929 19:06:46.457072 27453 leader_election.cc:288] T 00000000000000000000000000000000 P f782409ea8ad452ab278574c3bcc5c45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:06:46.457110 27456 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f782409ea8ad452ab278574c3bcc5c45 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:06:46.457280 27456 raft_consensus.cc:686] T 00000000000000000000000000000000 P f782409ea8ad452ab278574c3bcc5c45 [term 1 LEADER]: Becoming Leader. State: Replica: f782409ea8ad452ab278574c3bcc5c45, State: Running, Role: LEADER
I20260929 19:06:46.457314 27453 sys_catalog.cc:564] T 00000000000000000000000000000000 P f782409ea8ad452ab278574c3bcc5c45 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:06:46.457415 27456 consensus_queue.cc:237] T 00000000000000000000000000000000 P f782409ea8ad452ab278574c3bcc5c45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f782409ea8ad452ab278574c3bcc5c45" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 44925 } }
I20260929 19:06:46.457762 27458 sys_catalog.cc:455] T 00000000000000000000000000000000 P f782409ea8ad452ab278574c3bcc5c45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f782409ea8ad452ab278574c3bcc5c45. Latest consensus state: current_term: 1 leader_uuid: "f782409ea8ad452ab278574c3bcc5c45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f782409ea8ad452ab278574c3bcc5c45" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 44925 } } }
I20260929 19:06:46.457926 27458 sys_catalog.cc:458] T 00000000000000000000000000000000 P f782409ea8ad452ab278574c3bcc5c45 [sys.catalog]: This master's current role is: LEADER
I20260929 19:06:46.457840 27457 sys_catalog.cc:455] T 00000000000000000000000000000000 P f782409ea8ad452ab278574c3bcc5c45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f782409ea8ad452ab278574c3bcc5c45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f782409ea8ad452ab278574c3bcc5c45" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 44925 } } }
I20260929 19:06:46.458150 27457 sys_catalog.cc:458] T 00000000000000000000000000000000 P f782409ea8ad452ab278574c3bcc5c45 [sys.catalog]: This master's current role is: LEADER
I20260929 19:06:46.458408 27465 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:06:46.458899 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 27386
I20260929 19:06:46.458917 27465 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:06:46.459005 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:06:46.459213 8825 external_mini_cluster.cc:1428] Setting key 9ec44b328c6380191a4e4f88b5f43a53
I20260929 19:06:46.460511 27465 catalog_manager.cc:1347] Generated new cluster ID: 6be28f00f7db45fba77b9c3e12a24fae
I20260929 19:06:46.460563 27465 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:06:46.465348 27475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:46.459663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49244 (local address 127.8.158.126:44925)
0929 19:06:46.459871 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:46.459879 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:46.459931 (+ 52us) server_negotiation.cc:407] Connection header received
0929 19:06:46.460088 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:46.460095 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:46.460163 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:46.460546 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:46.460632 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:46.461983 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:46.463046 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:46.463683 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:46.463810 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:46.464515 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:46.464518 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:46.464519 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:06:46.464740 (+ 221us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:46.464782 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:06:46.464796 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:46.464891 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:46.464969 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 19:06:46.465147 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":89,"threads_started":1}
I20260929 19:06:46.469830 27465 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:06:46.470525 27465 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:06:46.474947 27465 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f782409ea8ad452ab278574c3bcc5c45: Generated new TSK 0
I20260929 19:06:46.475095 27465 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:06:46.477514 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.65:0
--local_ip_for_outbound_sockets=127.8.158.65
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:44925
--builtin_ntp_servers=127.8.158.84:43631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 19:06:46.555783 27477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:06:46.555971 27477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:06:46.555992 27477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:06:46.557197 27477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:06:46.557235 27477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:06:46.557260 27477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.65
I20260929 19:06:46.558825 27477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:43631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:44925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.65
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:06:46.559048 27477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:06:46.559350 27477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:06:46.565164 27483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:46.565361 27477 server_base.cc:1034] running on GCE node
W20260929 19:06:46.565176 27485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:46.565233 27482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:46.565704 27477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:06:46.565945 27477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:06:46.567116 27477 hybrid_clock.cc:648] HybridClock initialized: now 1790708806567095 us; error 36 us; skew 500 ppm
I20260929 19:06:46.568408 27477 webserver.cc:460] Webserver started at http://127.8.158.65:44949/ using document root <none> and password file <none>
I20260929 19:06:46.568648 27477 fs_manager.cc:359] Metadata directory not provided
I20260929 19:06:46.568714 27477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:06:46.568847 27477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:06:46.570029 27477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-0/data/instance:
uuid: "13cc00d514c34881812b4588ad494a67"
format_stamp: "Formatted at 2026-09-29 19:06:46 on dist-test-slave-nmsz"
server_key: "e918722fc1319d5d432883903bcadb6e"
server_key_iv: "3b212efd5edacf51011c04df4651b8f9"
server_key_version: "encryptionkey@0"
I20260929 19:06:46.570475 27477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance:
uuid: "13cc00d514c34881812b4588ad494a67"
format_stamp: "Formatted at 2026-09-29 19:06:46 on dist-test-slave-nmsz"
server_key: "e918722fc1319d5d432883903bcadb6e"
server_key_iv: "3b212efd5edacf51011c04df4651b8f9"
server_key_version: "encryptionkey@0"
I20260929 19:06:46.571800 27477 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:06:46.572539 27491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:46.572772 27477 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:06:46.572868 27477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-0/wal
uuid: "13cc00d514c34881812b4588ad494a67"
format_stamp: "Formatted at 2026-09-29 19:06:46 on dist-test-slave-nmsz"
server_key: "e918722fc1319d5d432883903bcadb6e"
server_key_iv: "3b212efd5edacf51011c04df4651b8f9"
server_key_version: "encryptionkey@0"
I20260929 19:06:46.572974 27477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:06:46.606470 27477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:06:46.606729 27477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:06:46.606832 27477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:06:46.607016 27477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:06:46.607362 27477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:06:46.607399 27477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:46.607424 27477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:06:46.607477 27477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:46.613139 27477 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.65:32871
I20260929 19:06:46.613200 27604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.65:32871 every 8 connection(s)
I20260929 19:06:46.613511 27477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
I20260929 19:06:46.618302 27475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:46.614213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:50581 (local address 127.8.158.126:44925)
0929 19:06:46.614289 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:46.614292 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:46.614545 (+ 253us) server_negotiation.cc:407] Connection header received
0929 19:06:46.615047 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:46.615048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:46.615061 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:46.615115 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:46.615739 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:46.616244 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:46.617155 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:46.617341 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:46.617405 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:46.618091 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:46.618092 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:46.618093 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:06:46.618097 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:46.618111 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:06:46.618115 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:46.618158 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:46.618224 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 19:06:46.618247 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 19:06:46.618515 27607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:46.614187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:44925 (local address 127.8.158.65:50581)
0929 19:06:46.614427 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:46.614459 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:46.614915 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:46.615161 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:46.615164 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:46.615172 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:06:46.615649 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:46.615653 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:46.616320 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:46.616320 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:46.617067 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:46.617068 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:46.617413 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:46.617414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:46.617519 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:46.617960 (+ 441us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:06:46.617961 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:06:46.617986 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:06:46.618007 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:06:46.618157 (+ 150us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:06:46.618159 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:06:46.618161 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 19:06:46.618205 (+ 44us) client_negotiation.cc:240] Negotiation successful
0929 19:06:46.618271 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":96,"threads_started":1}
I20260929 19:06:46.619016 27605 heartbeater.cc:347] Connected to a master server at 127.8.158.126:44925
I20260929 19:06:46.619108 27605 heartbeater.cc:464] Registering TS with master...
I20260929 19:06:46.619300 27605 heartbeater.cc:511] Master 127.8.158.126:44925 requested a full tablet report, sending...
I20260929 19:06:46.619783 27417 ts_manager.cc:194] Registered new tserver with Master: 13cc00d514c34881812b4588ad494a67 (127.8.158.65:32871)
I20260929 19:06:46.620388 27417 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.158.65:50581
I20260929 19:06:46.622303 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 27477
I20260929 19:06:46.622430 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance
I20260929 19:06:46.622634 8825 external_mini_cluster.cc:1428] Setting key c3325805eb1bb7776902a9ba11e0f144
I20260929 19:06:46.623418 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.66:0
--local_ip_for_outbound_sockets=127.8.158.66
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:44925
--builtin_ntp_servers=127.8.158.84:43631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 19:06:46.701853 27608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:06:46.702041 27608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:06:46.702062 27608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:06:46.703244 27608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:06:46.703284 27608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:06:46.703310 27608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.66
I20260929 19:06:46.704604 27608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:43631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:44925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.66
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:06:46.704830 27608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:06:46.705129 27608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:06:46.709854 27614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:46.709853 27613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:46.710032 27608 server_base.cc:1034] running on GCE node
W20260929 19:06:46.710105 27616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:46.710348 27608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:06:46.710577 27608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:06:46.711759 27608 hybrid_clock.cc:648] HybridClock initialized: now 1790708806711756 us; error 34 us; skew 500 ppm
I20260929 19:06:46.712972 27608 webserver.cc:460] Webserver started at http://127.8.158.66:43127/ using document root <none> and password file <none>
I20260929 19:06:46.713217 27608 fs_manager.cc:359] Metadata directory not provided
I20260929 19:06:46.713284 27608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:06:46.713414 27608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:06:46.714463 27608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-1/data/instance:
uuid: "ebb3e747c0c04293802fe1126c511a44"
format_stamp: "Formatted at 2026-09-29 19:06:46 on dist-test-slave-nmsz"
server_key: "f80305285e4e76b8055f3c1ca08a7ed7"
server_key_iv: "e78de221e532565dfb798356ba1f6007"
server_key_version: "encryptionkey@0"
I20260929 19:06:46.714793 27608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance:
uuid: "ebb3e747c0c04293802fe1126c511a44"
format_stamp: "Formatted at 2026-09-29 19:06:46 on dist-test-slave-nmsz"
server_key: "f80305285e4e76b8055f3c1ca08a7ed7"
server_key_iv: "e78de221e532565dfb798356ba1f6007"
server_key_version: "encryptionkey@0"
I20260929 19:06:46.716022 27608 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:06:46.716717 27622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:46.716894 27608 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:06:46.716987 27608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-1/wal
uuid: "ebb3e747c0c04293802fe1126c511a44"
format_stamp: "Formatted at 2026-09-29 19:06:46 on dist-test-slave-nmsz"
server_key: "f80305285e4e76b8055f3c1ca08a7ed7"
server_key_iv: "e78de221e532565dfb798356ba1f6007"
server_key_version: "encryptionkey@0"
I20260929 19:06:46.717082 27608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:06:46.737854 27608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:06:46.738125 27608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:06:46.738269 27608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:06:46.738505 27608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:06:46.738811 27608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:06:46.738866 27608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:46.738897 27608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:06:46.738922 27608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:46.744508 27608 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.66:38271
I20260929 19:06:46.744563 27735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.66:38271 every 8 connection(s)
I20260929 19:06:46.744868 27608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
I20260929 19:06:46.747789 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 27608
I20260929 19:06:46.747891 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance
I20260929 19:06:46.748086 8825 external_mini_cluster.cc:1428] Setting key d2292f0274645c922f7516368aa054fd
I20260929 19:06:46.749198 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.67:0
--local_ip_for_outbound_sockets=127.8.158.67
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:44925
--builtin_ntp_servers=127.8.158.84:43631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 19:06:46.750669 27475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:46.745559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:39965 (local address 127.8.158.126:44925)
0929 19:06:46.745603 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:46.745605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:46.745871 (+ 266us) server_negotiation.cc:407] Connection header received
0929 19:06:46.746439 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:46.746440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:46.746453 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:46.746551 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:46.747219 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:46.747816 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:46.748976 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:46.749291 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:46.749364 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:46.750291 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:46.750293 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:46.750293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:06:46.750297 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:46.750349 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:06:46.750354 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:46.750422 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:46.750521 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 19:06:46.750567 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 19:06:46.750880 27738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:46.745564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:44925 (local address 127.8.158.66:39965)
0929 19:06:46.745747 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:46.745777 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:46.746283 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:46.746623 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:46.746627 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:46.746639 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:06:46.747116 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:46.747120 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:46.747920 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:46.747921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:46.748911 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:46.748913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:46.749430 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:46.749432 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:46.749607 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:46.750137 (+ 530us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:06:46.750138 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:06:46.750174 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:06:46.750200 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:06:46.750428 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:06:46.750432 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:06:46.750435 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:06:46.750514 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 19:06:46.750629 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":84,"threads_started":1}
I20260929 19:06:46.751430 27736 heartbeater.cc:347] Connected to a master server at 127.8.158.126:44925
I20260929 19:06:46.751537 27736 heartbeater.cc:464] Registering TS with master...
I20260929 19:06:46.751705 27736 heartbeater.cc:511] Master 127.8.158.126:44925 requested a full tablet report, sending...
I20260929 19:06:46.752079 27417 ts_manager.cc:194] Registered new tserver with Master: ebb3e747c0c04293802fe1126c511a44 (127.8.158.66:38271)
I20260929 19:06:46.752593 27417 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.158.66:39965
W20260929 19:06:46.829017 27739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:06:46.829203 27739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:06:46.829224 27739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:06:46.830274 27739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:06:46.830307 27739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:06:46.830348 27739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.67
I20260929 19:06:46.831588 27739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:43631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:44925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.67
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:06:46.831760 27739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:06:46.831976 27739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:06:46.836985 27745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:46.836983 27744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:46.837031 27739 server_base.cc:1034] running on GCE node
W20260929 19:06:46.836985 27747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:46.837479 27739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:06:46.837723 27739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:06:46.838876 27739 hybrid_clock.cc:648] HybridClock initialized: now 1790708806838844 us; error 43 us; skew 500 ppm
I20260929 19:06:46.840063 27739 webserver.cc:460] Webserver started at http://127.8.158.67:40745/ using document root <none> and password file <none>
I20260929 19:06:46.840301 27739 fs_manager.cc:359] Metadata directory not provided
I20260929 19:06:46.840385 27739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:06:46.840531 27739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:06:46.841508 27739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-2/data/instance:
uuid: "2690de5ba4ec4f49a2c13be2d7d401e6"
format_stamp: "Formatted at 2026-09-29 19:06:46 on dist-test-slave-nmsz"
server_key: "d5fba2d96064b63c0b2106a7375f9516"
server_key_iv: "d0badd0ad2fb27f52fb798f4e754a3d3"
server_key_version: "encryptionkey@0"
I20260929 19:06:46.841848 27739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance:
uuid: "2690de5ba4ec4f49a2c13be2d7d401e6"
format_stamp: "Formatted at 2026-09-29 19:06:46 on dist-test-slave-nmsz"
server_key: "d5fba2d96064b63c0b2106a7375f9516"
server_key_iv: "d0badd0ad2fb27f52fb798f4e754a3d3"
server_key_version: "encryptionkey@0"
I20260929 19:06:46.843185 27739 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:06:46.843967 27753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:46.844163 27739 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:06:46.844260 27739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-2/wal
uuid: "2690de5ba4ec4f49a2c13be2d7d401e6"
format_stamp: "Formatted at 2026-09-29 19:06:46 on dist-test-slave-nmsz"
server_key: "d5fba2d96064b63c0b2106a7375f9516"
server_key_iv: "d0badd0ad2fb27f52fb798f4e754a3d3"
server_key_version: "encryptionkey@0"
I20260929 19:06:46.844348 27739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:06:46.859756 27739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:06:46.860020 27739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:06:46.860164 27739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:06:46.860375 27739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:06:46.860670 27739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:06:46.860723 27739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:46.860761 27739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:06:46.860788 27739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:46.866305 27739 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.67:42351
I20260929 19:06:46.866391 27866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.67:42351 every 8 connection(s)
I20260929 19:06:46.866706 27739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
I20260929 19:06:46.872041 27475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:46.867403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:41289 (local address 127.8.158.126:44925)
0929 19:06:46.867461 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:46.867463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:46.867721 (+ 258us) server_negotiation.cc:407] Connection header received
0929 19:06:46.868151 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:46.868152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:46.868163 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:46.868221 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:46.868817 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:46.869303 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:46.870475 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:46.870831 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:46.870905 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:46.871829 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:46.871830 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:46.871831 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:06:46.871835 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:46.871849 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:06:46.871853 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:46.871911 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:46.871961 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 19:06:46.871987 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 19:06:46.872280 27869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:46.867358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:44925 (local address 127.8.158.67:41289)
0929 19:06:46.867572 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:46.867611 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:46.868066 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:46.868268 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:46.868271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:46.868279 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:06:46.868739 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:46.868743 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:46.869404 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:46.869405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:46.870377 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:46.870383 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:46.870937 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:46.870938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:46.871112 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:46.871684 (+ 572us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:06:46.871684 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:06:46.871718 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:06:46.871739 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:06:46.871914 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:06:46.871916 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:06:46.871918 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 19:06:46.871961 (+ 43us) client_negotiation.cc:240] Negotiation successful
0929 19:06:46.872029 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":63,"threads_started":1}
I20260929 19:06:46.872757 27867 heartbeater.cc:347] Connected to a master server at 127.8.158.126:44925
I20260929 19:06:46.872861 27867 heartbeater.cc:464] Registering TS with master...
I20260929 19:06:46.873034 27867 heartbeater.cc:511] Master 127.8.158.126:44925 requested a full tablet report, sending...
I20260929 19:06:46.873354 27417 ts_manager.cc:194] Registered new tserver with Master: 2690de5ba4ec4f49a2c13be2d7d401e6 (127.8.158.67:42351)
I20260929 19:06:46.873667 27417 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.158.67:41289
I20260929 19:06:46.874763 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 27739
I20260929 19:06:46.874840 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance
I20260929 19:06:46.874991 8825 external_mini_cluster.cc:1428] Setting key ffd188f34a4e9c16210b2c8d1d75bf3c
I20260929 19:06:46.875793 8825 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:06:46.879868 27475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:46.877202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49254 (local address 127.8.158.126:44925)
0929 19:06:46.877266 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:46.877268 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:46.877313 (+ 45us) server_negotiation.cc:407] Connection header received
0929 19:06:46.877349 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:46.877350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:46.877359 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:46.877444 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:46.877577 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:46.878010 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:46.878742 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:46.878912 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:46.878964 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:46.879580 (+ 616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:46.879581 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:46.879581 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:06:46.879585 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:46.879596 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:06:46.879599 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:46.879660 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:46.879742 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 19:06:46.879772 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 19:06:46.881439 27417 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:06:46.883080 27417 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:06:46.889132 27877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:46.885039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.66:38271 (local address 127.0.0.1:47940)
0929 19:06:46.885284 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:46.885321 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:46.885397 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:46.885692 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:46.885694 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:46.885696 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:06:46.885762 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:46.885764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:46.887175 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:46.887176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:46.888058 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:46.888060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:46.888853 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:46.888854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:46.888925 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:46.888939 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 19:06:46.889015 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 19:06:46.889059 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":73,"thread_start_us":50,"threads_started":1}
I20260929 19:06:46.889146 27878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:46.885121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47940 (local address 127.8.158.66:38271)
0929 19:06:46.885339 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:46.885343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:46.885355 (+ 12us) server_negotiation.cc:407] Connection header received
0929 19:06:46.885508 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:46.885511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:46.885530 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:46.885613 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:06:46.885846 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:46.887066 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:46.888162 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:46.888772 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:46.888846 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:46.888924 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:46.889029 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 19:06:46.889066 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":110,"threads_started":1}
I20260929 19:06:46.889757 27875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:46.885057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:32871 (local address 127.0.0.1:40894)
0929 19:06:46.885812 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:46.885820 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:46.885888 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:46.886184 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:46.886185 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:46.886187 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:06:46.886225 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:46.886227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:46.887341 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:46.887342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:46.888633 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:46.888634 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:46.889522 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:46.889523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:46.889581 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:46.889585 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:06:46.889636 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 19:06:46.889676 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"mutex_wait_us":4,"thread_start_us":86,"threads_started":1}
I20260929 19:06:46.889757 27880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:46.885461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40894 (local address 127.8.158.65:32871)
0929 19:06:46.885637 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:46.885640 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:46.885883 (+ 243us) server_negotiation.cc:407] Connection header received
0929 19:06:46.886016 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:46.886018 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:46.886033 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:46.886105 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:06:46.886283 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:46.887248 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:46.888709 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:46.889448 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:46.889520 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:46.889599 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:46.889650 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 19:06:46.889682 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":84,"threads_started":1}
I20260929 19:06:46.889803 27670 tablet_service.cc:1467] Processing CreateTablet for tablet 4400c6896d2b41b28d56e408b1e96533 (DEFAULT_TABLE table=test-table [id=e84bf3198cab4e70b77baeb0af5a5f6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:06:46.889973 27879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:46.885276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50230 (local address 127.8.158.67:42351)
0929 19:06:46.885447 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:46.885452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:46.885639 (+ 187us) server_negotiation.cc:407] Connection header received
0929 19:06:46.885674 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:46.885676 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:46.885689 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:46.885756 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:06:46.885933 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:46.887071 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:46.888493 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:46.889330 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:46.889395 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:46.889472 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:46.889717 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 19:06:46.889756 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":85,"threads_started":1}
I20260929 19:06:46.890185 27670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4400c6896d2b41b28d56e408b1e96533. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:06:46.890267 27876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:46.885057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.67:42351 (local address 127.0.0.1:50230)
0929 19:06:46.885512 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:46.885520 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:46.885586 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:46.885776 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:46.885777 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:46.885779 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:06:46.885840 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:46.885841 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:46.887152 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:46.887154 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:46.888413 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:46.888416 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:46.889506 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:46.889507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:46.889570 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:46.889574 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:06:46.889622 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 19:06:46.889659 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":35,"thread_start_us":79,"threads_started":1}
I20260929 19:06:46.890537 27539 tablet_service.cc:1467] Processing CreateTablet for tablet 4400c6896d2b41b28d56e408b1e96533 (DEFAULT_TABLE table=test-table [id=e84bf3198cab4e70b77baeb0af5a5f6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:06:46.890844 27539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4400c6896d2b41b28d56e408b1e96533. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:06:46.891038 27801 tablet_service.cc:1467] Processing CreateTablet for tablet 4400c6896d2b41b28d56e408b1e96533 (DEFAULT_TABLE table=test-table [id=e84bf3198cab4e70b77baeb0af5a5f6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:06:46.891316 27801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4400c6896d2b41b28d56e408b1e96533. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:06:46.893234 27881 tablet_bootstrap.cc:492] T 4400c6896d2b41b28d56e408b1e96533 P 13cc00d514c34881812b4588ad494a67: Bootstrap starting.
I20260929 19:06:46.893697 27882 tablet_bootstrap.cc:492] T 4400c6896d2b41b28d56e408b1e96533 P ebb3e747c0c04293802fe1126c511a44: Bootstrap starting.
I20260929 19:06:46.893803 27881 tablet_bootstrap.cc:654] T 4400c6896d2b41b28d56e408b1e96533 P 13cc00d514c34881812b4588ad494a67: Neither blocks nor log segments found. Creating new log.
I20260929 19:06:46.894016 27881 log.cc:824] T 4400c6896d2b41b28d56e408b1e96533 P 13cc00d514c34881812b4588ad494a67: Log is configured to *not* fsync() on all Append() calls
I20260929 19:06:46.894294 27882 tablet_bootstrap.cc:654] T 4400c6896d2b41b28d56e408b1e96533 P ebb3e747c0c04293802fe1126c511a44: Neither blocks nor log segments found. Creating new log.
I20260929 19:06:46.894618 27882 log.cc:824] T 4400c6896d2b41b28d56e408b1e96533 P ebb3e747c0c04293802fe1126c511a44: Log is configured to *not* fsync() on all Append() calls
I20260929 19:06:46.894765 27881 tablet_bootstrap.cc:492] T 4400c6896d2b41b28d56e408b1e96533 P 13cc00d514c34881812b4588ad494a67: No bootstrap required, opened a new log
I20260929 19:06:46.894833 27881 ts_tablet_manager.cc:1397] T 4400c6896d2b41b28d56e408b1e96533 P 13cc00d514c34881812b4588ad494a67: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:06:46.895311 27882 tablet_bootstrap.cc:492] T 4400c6896d2b41b28d56e408b1e96533 P ebb3e747c0c04293802fe1126c511a44: No bootstrap required, opened a new log
I20260929 19:06:46.895392 27882 ts_tablet_manager.cc:1397] T 4400c6896d2b41b28d56e408b1e96533 P ebb3e747c0c04293802fe1126c511a44: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:06:46.896776 27881 raft_consensus.cc:348] T 4400c6896d2b41b28d56e408b1e96533 P 13cc00d514c34881812b4588ad494a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb3e747c0c04293802fe1126c511a44" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38271 } } peers { permanent_uuid: "2690de5ba4ec4f49a2c13be2d7d401e6" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 42351 } } peers { permanent_uuid: "13cc00d514c34881812b4588ad494a67" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 32871 } }
I20260929 19:06:46.896935 27881 raft_consensus.cc:374] T 4400c6896d2b41b28d56e408b1e96533 P 13cc00d514c34881812b4588ad494a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:06:46.896986 27881 raft_consensus.cc:729] T 4400c6896d2b41b28d56e408b1e96533 P 13cc00d514c34881812b4588ad494a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13cc00d514c34881812b4588ad494a67, State: Initialized, Role: FOLLOWER
I20260929 19:06:46.897097 27881 consensus_queue.cc:260] T 4400c6896d2b41b28d56e408b1e96533 P 13cc00d514c34881812b4588ad494a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb3e747c0c04293802fe1126c511a44" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38271 } } peers { permanent_uuid: "2690de5ba4ec4f49a2c13be2d7d401e6" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 42351 } } peers { permanent_uuid: "13cc00d514c34881812b4588ad494a67" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 32871 } }
I20260929 19:06:46.897341 27881 ts_tablet_manager.cc:1428] T 4400c6896d2b41b28d56e408b1e96533 P 13cc00d514c34881812b4588ad494a67: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 19:06:46.897392 27605 heartbeater.cc:503] Master 127.8.158.126:44925 was elected leader, sending a full tablet report...
I20260929 19:06:46.897481 27884 tablet_bootstrap.cc:492] T 4400c6896d2b41b28d56e408b1e96533 P 2690de5ba4ec4f49a2c13be2d7d401e6: Bootstrap starting.
I20260929 19:06:46.897504 27882 raft_consensus.cc:348] T 4400c6896d2b41b28d56e408b1e96533 P ebb3e747c0c04293802fe1126c511a44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb3e747c0c04293802fe1126c511a44" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38271 } } peers { permanent_uuid: "2690de5ba4ec4f49a2c13be2d7d401e6" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 42351 } } peers { permanent_uuid: "13cc00d514c34881812b4588ad494a67" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 32871 } }
I20260929 19:06:46.897696 27882 raft_consensus.cc:374] T 4400c6896d2b41b28d56e408b1e96533 P ebb3e747c0c04293802fe1126c511a44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:06:46.897739 27882 raft_consensus.cc:729] T 4400c6896d2b41b28d56e408b1e96533 P ebb3e747c0c04293802fe1126c511a44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebb3e747c0c04293802fe1126c511a44, State: Initialized, Role: FOLLOWER
I20260929 19:06:46.897848 27882 consensus_queue.cc:260] T 4400c6896d2b41b28d56e408b1e96533 P ebb3e747c0c04293802fe1126c511a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb3e747c0c04293802fe1126c511a44" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38271 } } peers { permanent_uuid: "2690de5ba4ec4f49a2c13be2d7d401e6" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 42351 } } peers { permanent_uuid: "13cc00d514c34881812b4588ad494a67" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 32871 } }
I20260929 19:06:46.898082 27882 ts_tablet_manager.cc:1428] T 4400c6896d2b41b28d56e408b1e96533 P ebb3e747c0c04293802fe1126c511a44: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 19:06:46.898113 27736 heartbeater.cc:503] Master 127.8.158.126:44925 was elected leader, sending a full tablet report...
I20260929 19:06:46.898337 27884 tablet_bootstrap.cc:654] T 4400c6896d2b41b28d56e408b1e96533 P 2690de5ba4ec4f49a2c13be2d7d401e6: Neither blocks nor log segments found. Creating new log.
I20260929 19:06:46.898675 27884 log.cc:824] T 4400c6896d2b41b28d56e408b1e96533 P 2690de5ba4ec4f49a2c13be2d7d401e6: Log is configured to *not* fsync() on all Append() calls
I20260929 19:06:46.899333 27884 tablet_bootstrap.cc:492] T 4400c6896d2b41b28d56e408b1e96533 P 2690de5ba4ec4f49a2c13be2d7d401e6: No bootstrap required, opened a new log
I20260929 19:06:46.899385 27884 ts_tablet_manager.cc:1397] T 4400c6896d2b41b28d56e408b1e96533 P 2690de5ba4ec4f49a2c13be2d7d401e6: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20260929 19:06:46.901152 27884 raft_consensus.cc:348] T 4400c6896d2b41b28d56e408b1e96533 P 2690de5ba4ec4f49a2c13be2d7d401e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb3e747c0c04293802fe1126c511a44" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38271 } } peers { permanent_uuid: "2690de5ba4ec4f49a2c13be2d7d401e6" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 42351 } } peers { permanent_uuid: "13cc00d514c34881812b4588ad494a67" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 32871 } }
I20260929 19:06:46.901294 27884 raft_consensus.cc:374] T 4400c6896d2b41b28d56e408b1e96533 P 2690de5ba4ec4f49a2c13be2d7d401e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:06:46.901357 27884 raft_consensus.cc:729] T 4400c6896d2b41b28d56e408b1e96533 P 2690de5ba4ec4f49a2c13be2d7d401e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2690de5ba4ec4f49a2c13be2d7d401e6, State: Initialized, Role: FOLLOWER
I20260929 19:06:46.901463 27884 consensus_queue.cc:260] T 4400c6896d2b41b28d56e408b1e96533 P 2690de5ba4ec4f49a2c13be2d7d401e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb3e747c0c04293802fe1126c511a44" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38271 } } peers { permanent_uuid: "2690de5ba4ec4f49a2c13be2d7d401e6" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 42351 } } peers { permanent_uuid: "13cc00d514c34881812b4588ad494a67" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 32871 } }
I20260929 19:06:46.901706 27884 ts_tablet_manager.cc:1428] T 4400c6896d2b41b28d56e408b1e96533 P 2690de5ba4ec4f49a2c13be2d7d401e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 19:06:46.901767 27867 heartbeater.cc:503] Master 127.8.158.126:44925 was elected leader, sending a full tablet report...
I20260929 19:06:46.949355 27887 raft_consensus.cc:482] T 4400c6896d2b41b28d56e408b1e96533 P ebb3e747c0c04293802fe1126c511a44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:06:46.949469 27887 raft_consensus.cc:504] T 4400c6896d2b41b28d56e408b1e96533 P ebb3e747c0c04293802fe1126c511a44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb3e747c0c04293802fe1126c511a44" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38271 } } peers { permanent_uuid: "2690de5ba4ec4f49a2c13be2d7d401e6" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 42351 } } peers { permanent_uuid: "13cc00d514c34881812b4588ad494a67" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 32871 } }
I20260929 19:06:46.949887 27887 leader_election.cc:288] T 4400c6896d2b41b28d56e408b1e96533 P ebb3e747c0c04293802fe1126c511a44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2690de5ba4ec4f49a2c13be2d7d401e6 (127.8.158.67:42351), 13cc00d514c34881812b4588ad494a67 (127.8.158.65:32871)
I20260929 19:06:46.952672 27879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:46.950155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:39049 (local address 127.8.158.67:42351)
0929 19:06:46.950212 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:46.950214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:46.950224 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:06:46.950254 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:46.950255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:46.950266 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:46.950374 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:06:46.950530 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:46.951033 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:46.951967 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:46.952334 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:46.952421 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:46.952429 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:46.952567 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 19:06:46.952604 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 19:06:46.952672 27738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:46.950056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.67:42351 (local address 127.8.158.66:39049)
0929 19:06:46.950129 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:46.950141 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:46.950185 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:46.950384 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:46.950385 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:46.950387 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:06:46.950457 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:46.950459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:46.951123 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:46.951124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:46.951867 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:46.951870 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:46.952419 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:46.952420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:46.952493 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:46.952496 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:06:46.952562 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 19:06:46.952604 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 19:06:46.952772 27880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:46.950139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:55309 (local address 127.8.158.65:32871)
0929 19:06:46.950186 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:46.950188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:46.950328 (+ 140us) server_negotiation.cc:407] Connection header received
0929 19:06:46.950419 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:46.950420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:46.950433 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:46.950506 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:06:46.950667 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:46.951117 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:46.952035 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:46.952398 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:46.952483 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:46.952490 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:46.952658 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 19:06:46.952700 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 19:06:46.952806 27890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:46.950079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:32871 (local address 127.8.158.66:55309)
0929 19:06:46.950247 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:46.950256 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:46.950364 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:46.950508 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:46.950509 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:46.950510 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:06:46.950589 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:46.950595 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:46.951200 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:46.951201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:46.951944 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:46.951946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:46.952482 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:46.952483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:46.952574 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:46.952577 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:06:46.952640 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 19:06:46.952679 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":116,"threads_started":1}
I20260929 19:06:46.953112 27821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4400c6896d2b41b28d56e408b1e96533" candidate_uuid: "ebb3e747c0c04293802fe1126c511a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2690de5ba4ec4f49a2c13be2d7d401e6" is_pre_election: true
I20260929 19:06:46.953146 27559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4400c6896d2b41b28d56e408b1e96533" candidate_uuid: "ebb3e747c0c04293802fe1126c511a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13cc00d514c34881812b4588ad494a67" is_pre_election: true
I20260929 19:06:46.953224 27821 raft_consensus.cc:2454] T 4400c6896d2b41b28d56e408b1e96533 P 2690de5ba4ec4f49a2c13be2d7d401e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebb3e747c0c04293802fe1126c511a44 in term 0.
I20260929 19:06:46.953310 27559 raft_consensus.cc:2454] T 4400c6896d2b41b28d56e408b1e96533 P 13cc00d514c34881812b4588ad494a67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebb3e747c0c04293802fe1126c511a44 in term 0.
I20260929 19:06:46.953449 27623 leader_election.cc:302] T 4400c6896d2b41b28d56e408b1e96533 P ebb3e747c0c04293802fe1126c511a44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2690de5ba4ec4f49a2c13be2d7d401e6, ebb3e747c0c04293802fe1126c511a44; no voters:
I20260929 19:06:46.953595 27887 raft_consensus.cc:2781] T 4400c6896d2b41b28d56e408b1e96533 P ebb3e747c0c04293802fe1126c511a44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:06:46.953666 27887 raft_consensus.cc:482] T 4400c6896d2b41b28d56e408b1e96533 P ebb3e747c0c04293802fe1126c511a44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:06:46.953702 27887 raft_consensus.cc:3037] T 4400c6896d2b41b28d56e408b1e96533 P ebb3e747c0c04293802fe1126c511a44 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:06:46.954525 27887 raft_consensus.cc:504] T 4400c6896d2b41b28d56e408b1e96533 P ebb3e747c0c04293802fe1126c511a44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb3e747c0c04293802fe1126c511a44" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38271 } } peers { permanent_uuid: "2690de5ba4ec4f49a2c13be2d7d401e6" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 42351 } } peers { permanent_uuid: "13cc00d514c34881812b4588ad494a67" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 32871 } }
I20260929 19:06:46.954667 27887 leader_election.cc:288] T 4400c6896d2b41b28d56e408b1e96533 P ebb3e747c0c04293802fe1126c511a44 [CANDIDATE]: Term 1 election: Requested vote from peers 2690de5ba4ec4f49a2c13be2d7d401e6 (127.8.158.67:42351), 13cc00d514c34881812b4588ad494a67 (127.8.158.65:32871)
I20260929 19:06:46.954843 27821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4400c6896d2b41b28d56e408b1e96533" candidate_uuid: "ebb3e747c0c04293802fe1126c511a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2690de5ba4ec4f49a2c13be2d7d401e6"
I20260929 19:06:46.954918 27821 raft_consensus.cc:3037] T 4400c6896d2b41b28d56e408b1e96533 P 2690de5ba4ec4f49a2c13be2d7d401e6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:06:46.955611 27821 raft_consensus.cc:2454] T 4400c6896d2b41b28d56e408b1e96533 P 2690de5ba4ec4f49a2c13be2d7d401e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebb3e747c0c04293802fe1126c511a44 in term 1.
I20260929 19:06:46.955703 27559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4400c6896d2b41b28d56e408b1e96533" candidate_uuid: "ebb3e747c0c04293802fe1126c511a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13cc00d514c34881812b4588ad494a67"
I20260929 19:06:46.955794 27623 leader_election.cc:302] T 4400c6896d2b41b28d56e408b1e96533 P ebb3e747c0c04293802fe1126c511a44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2690de5ba4ec4f49a2c13be2d7d401e6, ebb3e747c0c04293802fe1126c511a44; no voters:
I20260929 19:06:46.955802 27559 raft_consensus.cc:3037] T 4400c6896d2b41b28d56e408b1e96533 P 13cc00d514c34881812b4588ad494a67 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:06:46.955904 27887 raft_consensus.cc:2781] T 4400c6896d2b41b28d56e408b1e96533 P ebb3e747c0c04293802fe1126c511a44 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:06:46.956063 27887 raft_consensus.cc:686] T 4400c6896d2b41b28d56e408b1e96533 P ebb3e747c0c04293802fe1126c511a44 [term 1 LEADER]: Becoming Leader. State: Replica: ebb3e747c0c04293802fe1126c511a44, State: Running, Role: LEADER
I20260929 19:06:46.956168 27887 consensus_queue.cc:237] T 4400c6896d2b41b28d56e408b1e96533 P ebb3e747c0c04293802fe1126c511a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb3e747c0c04293802fe1126c511a44" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38271 } } peers { permanent_uuid: "2690de5ba4ec4f49a2c13be2d7d401e6" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 42351 } } peers { permanent_uuid: "13cc00d514c34881812b4588ad494a67" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 32871 } }
I20260929 19:06:46.956732 27559 raft_consensus.cc:2454] T 4400c6896d2b41b28d56e408b1e96533 P 13cc00d514c34881812b4588ad494a67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebb3e747c0c04293802fe1126c511a44 in term 1.
I20260929 19:06:46.957212 27417 catalog_manager.cc:5495] T 4400c6896d2b41b28d56e408b1e96533 P ebb3e747c0c04293802fe1126c511a44 reported cstate change: term changed from 0 to 1, leader changed from <none> to ebb3e747c0c04293802fe1126c511a44 (127.8.158.66). New cstate: current_term: 1 leader_uuid: "ebb3e747c0c04293802fe1126c511a44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebb3e747c0c04293802fe1126c511a44" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 38271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2690de5ba4ec4f49a2c13be2d7d401e6" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 42351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13cc00d514c34881812b4588ad494a67" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 32871 } health_report { overall_health: UNKNOWN } } }
I20260929 19:06:46.981940 27878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:46.979409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47956 (local address 127.8.158.66:38271)
0929 19:06:46.979464 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:46.979466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:46.979491 (+ 25us) server_negotiation.cc:407] Connection header received
0929 19:06:46.979617 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:46.979617 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:46.979638 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:46.979730 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:06:46.979911 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:46.980395 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:46.981137 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:46.981343 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:46.981436 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:46.981585 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:06:46.981684 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:06:46.981795 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:46.981834 (+ 39us) server_negotiation.cc:299] Negotiation successful
0929 19:06:46.981843 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 19:06:46.983788 27821 raft_consensus.cc:1261] T 4400c6896d2b41b28d56e408b1e96533 P 2690de5ba4ec4f49a2c13be2d7d401e6 [term 1 FOLLOWER]: Refusing update from remote peer ebb3e747c0c04293802fe1126c511a44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:06:46.983801 27559 raft_consensus.cc:1261] T 4400c6896d2b41b28d56e408b1e96533 P 13cc00d514c34881812b4588ad494a67 [term 1 FOLLOWER]: Refusing update from remote peer ebb3e747c0c04293802fe1126c511a44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:06:46.984181 27891 consensus_queue.cc:1035] T 4400c6896d2b41b28d56e408b1e96533 P ebb3e747c0c04293802fe1126c511a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13cc00d514c34881812b4588ad494a67" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 32871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:06:46.984330 27887 consensus_queue.cc:1035] T 4400c6896d2b41b28d56e408b1e96533 P ebb3e747c0c04293802fe1126c511a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2690de5ba4ec4f49a2c13be2d7d401e6" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 42351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:06:46.987236 27896 mvcc.cc:205] Tried to move back new op lower bound from 7334743273402691584 to 7334743273293103104. Current Snapshot: MvccSnapshot[applied={T|T < 7334743273402691584}]
I20260929 19:06:46.988808 27416 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:06:46.988888 27416 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:06:46.989660 27416 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f782409ea8ad452ab278574c3bcc5c45: Sending DeleteTablet for 3 replicas of tablet 4400c6896d2b41b28d56e408b1e96533
I20260929 19:06:46.989938 27670 tablet_service.cc:1514] Processing DeleteTablet for tablet 4400c6896d2b41b28d56e408b1e96533 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:06:46 UTC) from {username='slave'} at 127.0.0.1:47940
I20260929 19:06:46.989991 27801 tablet_service.cc:1514] Processing DeleteTablet for tablet 4400c6896d2b41b28d56e408b1e96533 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:06:46 UTC) from {username='slave'} at 127.0.0.1:50230
I20260929 19:06:46.990054 27539 tablet_service.cc:1514] Processing DeleteTablet for tablet 4400c6896d2b41b28d56e408b1e96533 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:06:46 UTC) from {username='slave'} at 127.0.0.1:40894
I20260929 19:06:46.990235 27906 tablet_replica.cc:330] T 4400c6896d2b41b28d56e408b1e96533 P 13cc00d514c34881812b4588ad494a67: stopping tablet replica
I20260929 19:06:46.990299 27904 tablet_replica.cc:330] T 4400c6896d2b41b28d56e408b1e96533 P ebb3e747c0c04293802fe1126c511a44: stopping tablet replica
I20260929 19:06:46.990353 27906 raft_consensus.cc:2229] T 4400c6896d2b41b28d56e408b1e96533 P 13cc00d514c34881812b4588ad494a67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:06:46.990391 27904 raft_consensus.cc:2229] T 4400c6896d2b41b28d56e408b1e96533 P ebb3e747c0c04293802fe1126c511a44 [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:06:46.990444 27906 raft_consensus.cc:2258] T 4400c6896d2b41b28d56e408b1e96533 P 13cc00d514c34881812b4588ad494a67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:06:46.990708 27904 raft_consensus.cc:2258] T 4400c6896d2b41b28d56e408b1e96533 P ebb3e747c0c04293802fe1126c511a44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:06:46.990816 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 27477
I20260929 19:06:46.991993 27905 tablet_replica.cc:330] T 4400c6896d2b41b28d56e408b1e96533 P 2690de5ba4ec4f49a2c13be2d7d401e6: stopping tablet replica
I20260929 19:06:46.992084 27905 raft_consensus.cc:2229] T 4400c6896d2b41b28d56e408b1e96533 P 2690de5ba4ec4f49a2c13be2d7d401e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:06:46.992192 27905 raft_consensus.cc:2258] T 4400c6896d2b41b28d56e408b1e96533 P 2690de5ba4ec4f49a2c13be2d7d401e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:06:46.992570 27904 ts_tablet_manager.cc:1905] T 4400c6896d2b41b28d56e408b1e96533 P ebb3e747c0c04293802fe1126c511a44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:06:46.992691 27905 ts_tablet_manager.cc:1905] T 4400c6896d2b41b28d56e408b1e96533 P 2690de5ba4ec4f49a2c13be2d7d401e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:06:46.994222 27905 ts_tablet_manager.cc:1918] T 4400c6896d2b41b28d56e408b1e96533 P 2690de5ba4ec4f49a2c13be2d7d401e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:06:46.994307 27905 log.cc:1196] T 4400c6896d2b41b28d56e408b1e96533 P 2690de5ba4ec4f49a2c13be2d7d401e6: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-2/wal/wals/4400c6896d2b41b28d56e408b1e96533
I20260929 19:06:46.994571 27904 ts_tablet_manager.cc:1918] T 4400c6896d2b41b28d56e408b1e96533 P ebb3e747c0c04293802fe1126c511a44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:06:46.994592 27905 ts_tablet_manager.cc:1939] T 4400c6896d2b41b28d56e408b1e96533 P 2690de5ba4ec4f49a2c13be2d7d401e6: Deleting consensus metadata
I20260929 19:06:46.994635 27904 log.cc:1196] T 4400c6896d2b41b28d56e408b1e96533 P ebb3e747c0c04293802fe1126c511a44: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708573736807-8825-0/minicluster-data/ts-1/wal/wals/4400c6896d2b41b28d56e408b1e96533
I20260929 19:06:46.994876 27904 ts_tablet_manager.cc:1939] T 4400c6896d2b41b28d56e408b1e96533 P ebb3e747c0c04293802fe1126c511a44: Deleting consensus metadata
I20260929 19:06:46.994951 27401 catalog_manager.cc:4841] TS 2690de5ba4ec4f49a2c13be2d7d401e6 (127.8.158.67:42351): tablet 4400c6896d2b41b28d56e408b1e96533 (table test-table [id=e84bf3198cab4e70b77baeb0af5a5f6f]) successfully deleted
I20260929 19:06:46.995227 27402 catalog_manager.cc:4841] TS ebb3e747c0c04293802fe1126c511a44 (127.8.158.66:38271): tablet 4400c6896d2b41b28d56e408b1e96533 (table test-table [id=e84bf3198cab4e70b77baeb0af5a5f6f]) successfully deleted
W20260929 19:06:46.995702 27404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.158.65:32871 (error 108)
W20260929 19:06:46.996610 27876 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:06:46.996452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:32871 (local address 127.0.0.1:40896)
0929 19:06:46.996525 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:46.996573 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.65:32871: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 19:06:46.996726 27404 catalog_manager.cc:4584] TS 13cc00d514c34881812b4588ad494a67 (127.8.158.65:32871): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4400c6896d2b41b28d56e408b1e96533: Network error: Client connection negotiation failed: client connection to 127.8.158.65:32871: connect: Connection reset by peer (error 104)
I20260929 19:06:46.996767 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 27608
I20260929 19:06:47.002002 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 27739
I20260929 19:06:47.006393 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 27386
2026-09-29T19:06:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (707 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:06:47 dist-test-slave-nmsz krb5kdc[27910](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:06:47 dist-test-slave-nmsz krb5kdc[27910](info): set up 2 sockets
Sep 29 19:06:47 dist-test-slave-nmsz krb5kdc[27910](info): commencing operation
krb5kdc: starting...
W20260929 19:06:48.030047 8825 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Sep 29 19:06:48 dist-test-slave-nmsz krb5kdc[27910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:06:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:06:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.158.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
I20260929 19:06:48.147436 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:42669
--webserver_interface=127.8.158.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:39683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:42669
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:06:48.226504 27923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:06:48.226708 27923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:06:48.226745 27923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:06:48.227958 27923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:06:48.228026 27923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:06:48.228052 27923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:06:48.228081 27923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:06:48.228103 27923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:06:48.229533 27923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:39683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:42669
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:42669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.158.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:06:48.229796 27923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:06:48.230011 27923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:06:48.234768 27928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:48.234781 27931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:48.234828 27923 server_base.cc:1034] running on GCE node
W20260929 19:06:48.234786 27929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:48.235251 27923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:06:48.235494 27923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:06:48.236630 27923 hybrid_clock.cc:648] HybridClock initialized: now 1790708808236622 us; error 30 us; skew 500 ppm
Sep 29 19:06:48 dist-test-slave-nmsz krb5kdc[27910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708808, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:06:48.239056 27923 init.cc:377] Logged in from keytab as kudu/127.8.158.126@KRBTEST.COM (short username kudu)
I20260929 19:06:48.239419 27923 webserver.cc:460] Webserver started at http://127.8.158.126:38223/ using document root <none> and password file <none>
I20260929 19:06:48.239655 27923 fs_manager.cc:359] Metadata directory not provided
I20260929 19:06:48.239724 27923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:06:48.239871 27923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:06:48.240862 27923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "a82b2135e94f49d8afa36ef69599b825"
format_stamp: "Formatted at 2026-09-29 19:06:48 on dist-test-slave-nmsz"
server_key: "38ae97801d1b6dbf274c4a0e2ebb73a8"
server_key_iv: "1736a5c07cf3a377abe477f559191698"
server_key_version: "encryptionkey@0"
I20260929 19:06:48.241227 27923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "a82b2135e94f49d8afa36ef69599b825"
format_stamp: "Formatted at 2026-09-29 19:06:48 on dist-test-slave-nmsz"
server_key: "38ae97801d1b6dbf274c4a0e2ebb73a8"
server_key_iv: "1736a5c07cf3a377abe477f559191698"
server_key_version: "encryptionkey@0"
I20260929 19:06:48.242568 27923 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:06:48.243268 27938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:48.243443 27923 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:06:48.243541 27923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "a82b2135e94f49d8afa36ef69599b825"
format_stamp: "Formatted at 2026-09-29 19:06:48 on dist-test-slave-nmsz"
server_key: "38ae97801d1b6dbf274c4a0e2ebb73a8"
server_key_iv: "1736a5c07cf3a377abe477f559191698"
server_key_version: "encryptionkey@0"
I20260929 19:06:48.243621 27923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:06:48.272666 27923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:06:48.275642 27923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:06:48.275802 27923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:06:48.280858 27923 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.126:42669
I20260929 19:06:48.280930 27990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.126:42669 every 8 connection(s)
I20260929 19:06:48.281183 27923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:06:48.281673 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 27923
I20260929 19:06:48.281795 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:06:48.282037 8825 external_mini_cluster.cc:1428] Setting key 1284bdaa373147950d66602404915982
I20260929 19:06:48.282145 27991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:06:48.285286 27991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a82b2135e94f49d8afa36ef69599b825: Bootstrap starting.
I20260929 19:06:48.285933 27991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a82b2135e94f49d8afa36ef69599b825: Neither blocks nor log segments found. Creating new log.
I20260929 19:06:48.286233 27991 log.cc:824] T 00000000000000000000000000000000 P a82b2135e94f49d8afa36ef69599b825: Log is configured to *not* fsync() on all Append() calls
I20260929 19:06:48.287055 27991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a82b2135e94f49d8afa36ef69599b825: No bootstrap required, opened a new log
Sep 29 19:06:48 dist-test-slave-nmsz krb5kdc[27910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:06:48.289021 27991 raft_consensus.cc:348] T 00000000000000000000000000000000 P a82b2135e94f49d8afa36ef69599b825 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a82b2135e94f49d8afa36ef69599b825" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 42669 } }
I20260929 19:06:48.289144 27991 raft_consensus.cc:374] T 00000000000000000000000000000000 P a82b2135e94f49d8afa36ef69599b825 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:06:48.289166 27991 raft_consensus.cc:729] T 00000000000000000000000000000000 P a82b2135e94f49d8afa36ef69599b825 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a82b2135e94f49d8afa36ef69599b825, State: Initialized, Role: FOLLOWER
I20260929 19:06:48.289305 27991 consensus_queue.cc:260] T 00000000000000000000000000000000 P a82b2135e94f49d8afa36ef69599b825 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a82b2135e94f49d8afa36ef69599b825" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 42669 } }
I20260929 19:06:48.289371 27991 raft_consensus.cc:388] T 00000000000000000000000000000000 P a82b2135e94f49d8afa36ef69599b825 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:06:48.289393 27991 raft_consensus.cc:482] T 00000000000000000000000000000000 P a82b2135e94f49d8afa36ef69599b825 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:06:48.289460 27991 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a82b2135e94f49d8afa36ef69599b825 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:06:48.290036 27991 raft_consensus.cc:504] T 00000000000000000000000000000000 P a82b2135e94f49d8afa36ef69599b825 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a82b2135e94f49d8afa36ef69599b825" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 42669 } }
I20260929 19:06:48.290143 27991 leader_election.cc:302] T 00000000000000000000000000000000 P a82b2135e94f49d8afa36ef69599b825 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a82b2135e94f49d8afa36ef69599b825; no voters:
I20260929 19:06:48.290293 27991 leader_election.cc:288] T 00000000000000000000000000000000 P a82b2135e94f49d8afa36ef69599b825 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:06:48.290374 27996 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a82b2135e94f49d8afa36ef69599b825 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:06:48.290498 27996 raft_consensus.cc:686] T 00000000000000000000000000000000 P a82b2135e94f49d8afa36ef69599b825 [term 1 LEADER]: Becoming Leader. State: Replica: a82b2135e94f49d8afa36ef69599b825, State: Running, Role: LEADER
I20260929 19:06:48.290596 27991 sys_catalog.cc:564] T 00000000000000000000000000000000 P a82b2135e94f49d8afa36ef69599b825 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:06:48.290593 27996 consensus_queue.cc:237] T 00000000000000000000000000000000 P a82b2135e94f49d8afa36ef69599b825 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a82b2135e94f49d8afa36ef69599b825" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 42669 } }
I20260929 19:06:48.290968 27998 sys_catalog.cc:455] T 00000000000000000000000000000000 P a82b2135e94f49d8afa36ef69599b825 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a82b2135e94f49d8afa36ef69599b825. Latest consensus state: current_term: 1 leader_uuid: "a82b2135e94f49d8afa36ef69599b825" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a82b2135e94f49d8afa36ef69599b825" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 42669 } } }
I20260929 19:06:48.291064 27998 sys_catalog.cc:458] T 00000000000000000000000000000000 P a82b2135e94f49d8afa36ef69599b825 [sys.catalog]: This master's current role is: LEADER
I20260929 19:06:48.291589 28004 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:06:48.292094 28004 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:06:48.293133 27997 sys_catalog.cc:455] T 00000000000000000000000000000000 P a82b2135e94f49d8afa36ef69599b825 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a82b2135e94f49d8afa36ef69599b825" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a82b2135e94f49d8afa36ef69599b825" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 42669 } } }
I20260929 19:06:48.293215 27997 sys_catalog.cc:458] T 00000000000000000000000000000000 P a82b2135e94f49d8afa36ef69599b825 [sys.catalog]: This master's current role is: LEADER
I20260929 19:06:48.293386 27994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:48.282627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40816 (local address 127.8.158.126:42669)
0929 19:06:48.282778 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:48.282785 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:48.282809 (+ 24us) server_negotiation.cc:407] Connection header received
0929 19:06:48.283262 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:48.283269 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:48.283313 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:48.283419 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:48.284182 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:48.285074 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:48.286108 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:48.286669 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:48.286789 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:48.289465 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:48.289469 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:48.289469 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:06:48.289481 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:48.291667 (+ 2186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:48.292107 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:48.292110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:48.292110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:48.292234 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:48.292447 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:48.292448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:48.292448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:48.292717 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:48.292844 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:48.293012 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 19:06:48.293154 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":73,"threads_started":1}
I20260929 19:06:48.295342 28004 catalog_manager.cc:1347] Generated new cluster ID: 3c05b3b0c7294d929db0a82cc99ebcd5
I20260929 19:06:48.295456 28004 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:06:48.312165 28004 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:06:48.312690 28004 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:06:48.321090 28004 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a82b2135e94f49d8afa36ef69599b825: Generated new TSK 0
I20260929 19:06:48.321269 28004 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.158.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab.
Entry for principal kudu/127.8.158.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab.
I20260929 19:06:48.347256 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.65:0
--local_ip_for_outbound_sockets=127.8.158.65
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:42669
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab
--principal=kudu/127.8.158.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:39683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:06:48.425930 28017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:06:48.426116 28017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:06:48.426139 28017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:06:48.427264 28017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:06:48.427305 28017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:06:48.427331 28017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.65
I20260929 19:06:48.428650 28017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:39683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.65.keytab
--principal=kudu/127.8.158.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:42669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.65
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:06:48.428814 28017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:06:48.428970 28017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:06:48.433765 28025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:48.433743 28022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:48.433784 28023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:48.433992 28017 server_base.cc:1034] running on GCE node
I20260929 19:06:48.434208 28017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:06:48.434475 28017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:06:48.435627 28017 hybrid_clock.cc:648] HybridClock initialized: now 1790708808435604 us; error 38 us; skew 500 ppm
Sep 29 19:06:48 dist-test-slave-nmsz krb5kdc[27910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708808, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:06:48.438030 28017 init.cc:377] Logged in from keytab as kudu/127.8.158.65@KRBTEST.COM (short username kudu)
I20260929 19:06:48.438519 28017 webserver.cc:460] Webserver started at http://127.8.158.65:38095/ using document root <none> and password file <none>
I20260929 19:06:48.438740 28017 fs_manager.cc:359] Metadata directory not provided
I20260929 19:06:48.438808 28017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:06:48.438946 28017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:06:48.439945 28017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-0/data/instance:
uuid: "13dfaa275cba432cb54449dd5f0eac83"
format_stamp: "Formatted at 2026-09-29 19:06:48 on dist-test-slave-nmsz"
server_key: "3a4f385dd120b63a11b478235cfed2b4"
server_key_iv: "82b2b4cbe5eaacd73e0eab4f807bb066"
server_key_version: "encryptionkey@0"
I20260929 19:06:48.440300 28017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance:
uuid: "13dfaa275cba432cb54449dd5f0eac83"
format_stamp: "Formatted at 2026-09-29 19:06:48 on dist-test-slave-nmsz"
server_key: "3a4f385dd120b63a11b478235cfed2b4"
server_key_iv: "82b2b4cbe5eaacd73e0eab4f807bb066"
server_key_version: "encryptionkey@0"
I20260929 19:06:48.441668 28017 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:06:48.442440 28032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:48.442617 28017 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:06:48.442711 28017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-0/wal
uuid: "13dfaa275cba432cb54449dd5f0eac83"
format_stamp: "Formatted at 2026-09-29 19:06:48 on dist-test-slave-nmsz"
server_key: "3a4f385dd120b63a11b478235cfed2b4"
server_key_iv: "82b2b4cbe5eaacd73e0eab4f807bb066"
server_key_version: "encryptionkey@0"
I20260929 19:06:48.442797 28017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:06:48.457576 28017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:06:48.459478 28017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:06:48.459635 28017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:06:48.459865 28017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:06:48.460166 28017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:06:48.460220 28017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:48.460261 28017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:06:48.460289 28017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:48.465692 28017 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.65:40473
I20260929 19:06:48.465758 28145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.65:40473 every 8 connection(s)
I20260929 19:06:48.466081 28017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
I20260929 19:06:48.471731 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 28017
Sep 29 19:06:48 dist-test-slave-nmsz krb5kdc[27910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708808, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.65@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:06:48.471860 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance
I20260929 19:06:48.472069 8825 external_mini_cluster.cc:1428] Setting key 10651277fb0a9c103b9e520976d4f89e
I20260929 19:06:48.475277 27994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:48.466828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:35723 (local address 127.8.158.126:42669)
0929 19:06:48.466900 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:48.466902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:48.467089 (+ 187us) server_negotiation.cc:407] Connection header received
0929 19:06:48.467546 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:48.467548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:48.467558 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:48.467619 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:48.468510 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:48.469018 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:48.469969 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:48.470165 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:48.470221 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:48.472419 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:48.472421 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:48.472421 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:06:48.472430 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:48.473835 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:48.474310 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:48.474311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:48.474311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:48.474403 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:48.474639 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:48.474640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:48.474640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:48.474779 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:48.474851 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:48.475102 (+ 251us) server_negotiation.cc:299] Negotiation successful
0929 19:06:48.475215 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 19:06:48.475512 28148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:48.466794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:42669 (local address 127.8.158.65:35723)
0929 19:06:48.466963 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:48.467003 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:48.467460 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:48.467694 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:48.467697 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:48.468237 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:06:48.468420 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:48.468424 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:48.469104 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:48.469105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:48.469879 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:48.469881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:48.470241 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:48.470241 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:48.470402 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:48.470873 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:06:48.470881 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:06:48.472292 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:06:48.473957 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:06:48.473965 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:06:48.473967 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:06:48.474238 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:06:48.474473 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:06:48.474473 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:06:48.474474 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:06:48.474573 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:06:48.474843 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:06:48.474847 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:06:48.474951 (+ 104us) client_negotiation.cc:769] Sending connection context
0929 19:06:48.475078 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 19:06:48.475245 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":104,"threads_started":1}
I20260929 19:06:48.475983 28146 heartbeater.cc:347] Connected to a master server at 127.8.158.126:42669
I20260929 19:06:48.476090 28146 heartbeater.cc:464] Registering TS with master...
I20260929 19:06:48.476260 28146 heartbeater.cc:511] Master 127.8.158.126:42669 requested a full tablet report, sending...
I20260929 19:06:48.476778 27955 ts_manager.cc:194] Registered new tserver with Master: 13dfaa275cba432cb54449dd5f0eac83 (127.8.158.65:40473)
I20260929 19:06:48.477361 27955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.65@KRBTEST.COM'} at 127.8.158.65:35723
WARNING: no policy specified for kudu/127.8.158.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab.
Entry for principal kudu/127.8.158.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab.
I20260929 19:06:48.496793 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.66:0
--local_ip_for_outbound_sockets=127.8.158.66
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:42669
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab
--principal=kudu/127.8.158.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:39683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:06:48.575662 28151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:06:48.575852 28151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:06:48.575875 28151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:06:48.577041 28151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:06:48.577083 28151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:06:48.577109 28151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.66
I20260929 19:06:48.578548 28151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:39683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.66.keytab
--principal=kudu/127.8.158.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:42669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.66
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:06:48.578775 28151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:06:48.579030 28151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:06:48.583878 28157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:48.583854 28159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:48.583855 28156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:48.584079 28151 server_base.cc:1034] running on GCE node
I20260929 19:06:48.584380 28151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:06:48.584617 28151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:06:48.585785 28151 hybrid_clock.cc:648] HybridClock initialized: now 1790708808585786 us; error 41 us; skew 500 ppm
Sep 29 19:06:48 dist-test-slave-nmsz krb5kdc[27910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708808, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:06:48.588307 28151 init.cc:377] Logged in from keytab as kudu/127.8.158.66@KRBTEST.COM (short username kudu)
I20260929 19:06:48.588660 28151 webserver.cc:460] Webserver started at http://127.8.158.66:44835/ using document root <none> and password file <none>
I20260929 19:06:48.588903 28151 fs_manager.cc:359] Metadata directory not provided
I20260929 19:06:48.588955 28151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:06:48.589078 28151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:06:48.590044 28151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-1/data/instance:
uuid: "ea4f0422c2f5493b8eab7e3ef6686aa6"
format_stamp: "Formatted at 2026-09-29 19:06:48 on dist-test-slave-nmsz"
server_key: "e9ea624ebc9f63a0fac7af80569698f7"
server_key_iv: "123ff64bd36183c26ced016798b230db"
server_key_version: "encryptionkey@0"
I20260929 19:06:48.590468 28151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance:
uuid: "ea4f0422c2f5493b8eab7e3ef6686aa6"
format_stamp: "Formatted at 2026-09-29 19:06:48 on dist-test-slave-nmsz"
server_key: "e9ea624ebc9f63a0fac7af80569698f7"
server_key_iv: "123ff64bd36183c26ced016798b230db"
server_key_version: "encryptionkey@0"
I20260929 19:06:48.591871 28151 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:06:48.592730 28166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:48.592914 28151 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:06:48.593042 28151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-1/wal
uuid: "ea4f0422c2f5493b8eab7e3ef6686aa6"
format_stamp: "Formatted at 2026-09-29 19:06:48 on dist-test-slave-nmsz"
server_key: "e9ea624ebc9f63a0fac7af80569698f7"
server_key_iv: "123ff64bd36183c26ced016798b230db"
server_key_version: "encryptionkey@0"
I20260929 19:06:48.593127 28151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:06:48.605734 28151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:06:48.607944 28151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:06:48.608116 28151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:06:48.608331 28151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:06:48.608631 28151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:06:48.608685 28151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:48.608726 28151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:06:48.608753 28151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:48.614275 28151 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.66:46783
I20260929 19:06:48.614367 28279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.66:46783 every 8 connection(s)
I20260929 19:06:48.614663 28151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
Sep 29 19:06:48 dist-test-slave-nmsz krb5kdc[27910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708808, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.66@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:06:48.620971 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 28151
I20260929 19:06:48.621114 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance
I20260929 19:06:48.621316 8825 external_mini_cluster.cc:1428] Setting key c3c0486496b5498ad0ed85aa7cbcb2dd
I20260929 19:06:48.623576 27994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:48.615407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:38135 (local address 127.8.158.126:42669)
0929 19:06:48.615474 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:48.615476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:48.615664 (+ 188us) server_negotiation.cc:407] Connection header received
0929 19:06:48.616106 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:48.616107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:48.616117 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:48.616183 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:48.617001 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:48.617480 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:48.618486 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:48.618667 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:48.618724 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:48.620663 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:48.620664 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:48.620665 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:06:48.620672 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:48.622235 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:48.622653 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:48.622654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:48.622655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:48.622711 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:48.622962 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:48.622962 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:48.622963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:48.623079 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:48.623152 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:48.623411 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 19:06:48.623490 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 19:06:48.623824 28282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:48.615385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:42669 (local address 127.8.158.66:38135)
0929 19:06:48.615552 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:48.615583 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:48.616004 (+ 421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:48.616235 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:48.616238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:48.616757 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:06:48.616928 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:48.616932 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:48.617558 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:48.617559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:48.618406 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:48.618409 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:48.618738 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:48.618739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:48.618830 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:48.619226 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:06:48.619233 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:06:48.620562 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:06:48.622333 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:06:48.622340 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:06:48.622342 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:06:48.622574 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:06:48.622785 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:06:48.622786 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:06:48.622786 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:06:48.622888 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:06:48.623144 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:06:48.623150 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:06:48.623278 (+ 128us) client_negotiation.cc:769] Sending connection context
0929 19:06:48.623379 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 19:06:48.623526 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
I20260929 19:06:48.624269 28280 heartbeater.cc:347] Connected to a master server at 127.8.158.126:42669
I20260929 19:06:48.624357 28280 heartbeater.cc:464] Registering TS with master...
I20260929 19:06:48.624536 28280 heartbeater.cc:511] Master 127.8.158.126:42669 requested a full tablet report, sending...
I20260929 19:06:48.624929 27955 ts_manager.cc:194] Registered new tserver with Master: ea4f0422c2f5493b8eab7e3ef6686aa6 (127.8.158.66:46783)
I20260929 19:06:48.625304 27955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.66@KRBTEST.COM'} at 127.8.158.66:38135
WARNING: no policy specified for kudu/127.8.158.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab.
Entry for principal kudu/127.8.158.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab.
I20260929 19:06:48.645890 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.67:0
--local_ip_for_outbound_sockets=127.8.158.67
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:42669
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab
--principal=kudu/127.8.158.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:39683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:06:48.724438 28285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:06:48.724624 28285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:06:48.724646 28285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:06:48.725782 28285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:06:48.725824 28285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:06:48.725850 28285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.67
I20260929 19:06:48.727313 28285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:39683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.67.keytab
--principal=kudu/127.8.158.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:42669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.67
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:06:48.727551 28285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:06:48.727772 28285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:06:48.732548 28290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:48.732533 28291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:48.732522 28293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:48.732751 28285 server_base.cc:1034] running on GCE node
I20260929 19:06:48.732960 28285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:06:48.733186 28285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:06:48.734406 28285 hybrid_clock.cc:648] HybridClock initialized: now 1790708808734392 us; error 41 us; skew 500 ppm
Sep 29 19:06:48 dist-test-slave-nmsz krb5kdc[27910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708808, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:06:48.736840 28285 init.cc:377] Logged in from keytab as kudu/127.8.158.67@KRBTEST.COM (short username kudu)
I20260929 19:06:48.737262 28285 webserver.cc:460] Webserver started at http://127.8.158.67:35161/ using document root <none> and password file <none>
I20260929 19:06:48.737479 28285 fs_manager.cc:359] Metadata directory not provided
I20260929 19:06:48.737547 28285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:06:48.737695 28285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:06:48.738785 28285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-2/data/instance:
uuid: "a0346a7a77c740eeb8b7a55d0b838a2f"
format_stamp: "Formatted at 2026-09-29 19:06:48 on dist-test-slave-nmsz"
server_key: "e11378a94a8901f6bf9f1a1fb8c4087e"
server_key_iv: "da4b40741c161cf79db3f4cc9231f697"
server_key_version: "encryptionkey@0"
I20260929 19:06:48.739248 28285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance:
uuid: "a0346a7a77c740eeb8b7a55d0b838a2f"
format_stamp: "Formatted at 2026-09-29 19:06:48 on dist-test-slave-nmsz"
server_key: "e11378a94a8901f6bf9f1a1fb8c4087e"
server_key_iv: "da4b40741c161cf79db3f4cc9231f697"
server_key_version: "encryptionkey@0"
I20260929 19:06:48.740653 28285 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:06:48.741528 28300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:48.741870 28285 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:06:48.741981 28285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-2/wal
uuid: "a0346a7a77c740eeb8b7a55d0b838a2f"
format_stamp: "Formatted at 2026-09-29 19:06:48 on dist-test-slave-nmsz"
server_key: "e11378a94a8901f6bf9f1a1fb8c4087e"
server_key_iv: "da4b40741c161cf79db3f4cc9231f697"
server_key_version: "encryptionkey@0"
I20260929 19:06:48.742054 28285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:06:48.754091 28285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:06:48.755893 28285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:06:48.756057 28285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:06:48.756273 28285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:06:48.756564 28285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:06:48.756626 28285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:48.756666 28285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:06:48.756693 28285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:48.762396 28285 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.67:36237
I20260929 19:06:48.762447 28413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.67:36237 every 8 connection(s)
I20260929 19:06:48.762733 28285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
Sep 29 19:06:48 dist-test-slave-nmsz krb5kdc[27910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708808, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.67@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:06:48.770572 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 28285
I20260929 19:06:48.770689 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance
I20260929 19:06:48.770884 8825 external_mini_cluster.cc:1428] Setting key cb39528360a32bdc95b5303592ee2254
I20260929 19:06:48.771652 27994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:48.763374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:43357 (local address 127.8.158.126:42669)
0929 19:06:48.763439 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:48.763441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:48.763625 (+ 184us) server_negotiation.cc:407] Connection header received
0929 19:06:48.764095 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:48.764097 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:48.764106 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:48.764166 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:48.765061 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:48.765549 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:48.766635 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:48.766861 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:48.766924 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:48.768913 (+ 1989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:48.768914 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:48.768914 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:06:48.768921 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:48.770094 (+ 1173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:48.770521 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:48.770523 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:48.770523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:48.770590 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:48.770963 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:48.770964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:48.770964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:48.771110 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:48.771198 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:48.771485 (+ 287us) server_negotiation.cc:299] Negotiation successful
0929 19:06:48.771566 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 19:06:48.771816 28416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:48.763322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:42669 (local address 127.8.158.67:43357)
0929 19:06:48.763489 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:48.763527 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:48.764006 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:48.764235 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:48.764238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:48.764776 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:06:48.764977 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:48.764981 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:48.765623 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:48.765624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:48.766546 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:48.766549 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:48.766934 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:48.766935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:48.767034 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:48.767516 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:06:48.767524 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:06:48.768825 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:06:48.770163 (+ 1338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:06:48.770169 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:06:48.770170 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:06:48.770446 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:06:48.770698 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:06:48.770699 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:06:48.770699 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:06:48.770846 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:06:48.771200 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:06:48.771207 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:06:48.771338 (+ 131us) client_negotiation.cc:769] Sending connection context
0929 19:06:48.771447 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 19:06:48.771595 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":92,"threads_started":1}
I20260929 19:06:48.772341 28414 heartbeater.cc:347] Connected to a master server at 127.8.158.126:42669
I20260929 19:06:48.772464 28414 heartbeater.cc:464] Registering TS with master...
I20260929 19:06:48.772639 28414 heartbeater.cc:511] Master 127.8.158.126:42669 requested a full tablet report, sending...
I20260929 19:06:48.773088 27955 ts_manager.cc:194] Registered new tserver with Master: a0346a7a77c740eeb8b7a55d0b838a2f (127.8.158.67:36237)
I20260929 19:06:48.773505 27955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.158.67@KRBTEST.COM'} at 127.8.158.67:43357
I20260929 19:06:48.783007 8825 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:06:48.790236 27994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:48.784326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40824 (local address 127.8.158.126:42669)
0929 19:06:48.784405 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:48.784406 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:48.784520 (+ 114us) server_negotiation.cc:407] Connection header received
0929 19:06:48.784584 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:48.784585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:48.784596 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:48.784663 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:48.785228 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:48.785699 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:48.786458 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:48.786667 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:48.786754 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:48.787906 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:48.787907 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:48.787908 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:06:48.787914 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:48.789066 (+ 1152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:48.789418 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:48.789419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:48.789420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:48.789477 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:48.789696 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:48.789697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:48.789697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:48.789846 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:48.789955 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:48.790059 (+ 104us) server_negotiation.cc:299] Negotiation successful
0929 19:06:48.790139 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 19:06:48.791989 27955 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40824:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:06:48.793516 27955 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:06:48.799556 28423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:48.795721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:40473 (local address 127.0.0.1:41528)
0929 19:06:48.795902 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:48.795943 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:48.796008 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:48.796365 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:48.796368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:48.796377 (+ 9us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:06:48.796442 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:48.796444 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:48.797468 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:48.797469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:48.798415 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:48.798418 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:48.799335 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:48.799336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:48.799413 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:48.799435 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 19:06:48.799471 (+ 36us) client_negotiation.cc:240] Negotiation successful
0929 19:06:48.799506 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":2,"thread_start_us":51,"threads_started":1}
I20260929 19:06:48.799912 28425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:48.796018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41528 (local address 127.8.158.65:40473)
0929 19:06:48.796193 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:48.796198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:48.796208 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:06:48.796247 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:48.796249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:48.796266 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:48.796346 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:06:48.796507 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:48.797361 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:48.798508 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:48.799233 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:48.799694 (+ 461us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:48.799771 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:48.799823 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 19:06:48.799856 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":106,"threads_started":1}
I20260929 19:06:48.800307 28422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:48.795969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.67:36237 (local address 127.0.0.1:44072)
0929 19:06:48.796047 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:48.796056 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:48.796104 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:48.796782 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:48.796783 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:48.796785 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:06:48.796846 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:48.796847 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:48.798113 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:48.798114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:48.799092 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:48.799095 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:48.800100 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:48.800101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:48.800165 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:48.800169 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:06:48.800199 (+ 30us) client_negotiation.cc:240] Negotiation successful
0929 19:06:48.800236 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":38,"thread_start_us":38,"threads_started":1}
I20260929 19:06:48.800544 28080 tablet_service.cc:1467] Processing CreateTablet for tablet 5ee3b0e82bbe403eb3b9c8e4c5de24ba (DEFAULT_TABLE table=test-table [id=8aaf16b1bb6743baa610b7fc84b5120f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:06:48.800766 28427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:48.796465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44072 (local address 127.8.158.67:36237)
0929 19:06:48.796639 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:48.796642 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:48.796651 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:06:48.796684 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:48.796686 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:48.796700 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:48.796783 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:06:48.796926 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:48.797986 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:48.799192 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:48.800002 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:48.800441 (+ 439us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:48.800499 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:48.800525 (+ 26us) server_negotiation.cc:299] Negotiation successful
0929 19:06:48.800566 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":86,"threads_started":1}
I20260929 19:06:48.800868 28424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:48.795785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.66:46783 (local address 127.0.0.1:35308)
0929 19:06:48.796041 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:48.796052 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:48.796104 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:48.796793 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:48.796794 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:48.796796 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:06:48.796842 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:48.796843 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:48.798148 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:48.798149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:48.799304 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:48.799307 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:48.800637 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:48.800638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:48.800700 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:48.800703 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:06:48.800762 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 19:06:48.800802 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":25,"thread_start_us":62,"threads_started":1}
I20260929 19:06:48.800944 28080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ee3b0e82bbe403eb3b9c8e4c5de24ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:06:48.801152 28426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:48.796228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35308 (local address 127.8.158.66:46783)
0929 19:06:48.796608 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:48.796613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:48.796624 (+ 11us) server_negotiation.cc:407] Connection header received
0929 19:06:48.796664 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:48.796666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:48.796682 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:48.796934 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:06:48.797004 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:48.798046 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:48.799385 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:48.800555 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:48.800625 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:48.800674 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:48.801020 (+ 346us) server_negotiation.cc:299] Negotiation successful
0929 19:06:48.801079 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":93,"threads_started":1}
I20260929 19:06:48.801360 28348 tablet_service.cc:1467] Processing CreateTablet for tablet 5ee3b0e82bbe403eb3b9c8e4c5de24ba (DEFAULT_TABLE table=test-table [id=8aaf16b1bb6743baa610b7fc84b5120f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:06:48.801671 28348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ee3b0e82bbe403eb3b9c8e4c5de24ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:06:48.801736 28214 tablet_service.cc:1467] Processing CreateTablet for tablet 5ee3b0e82bbe403eb3b9c8e4c5de24ba (DEFAULT_TABLE table=test-table [id=8aaf16b1bb6743baa610b7fc84b5120f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:06:48.802022 28214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ee3b0e82bbe403eb3b9c8e4c5de24ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:06:48.803610 28428 tablet_bootstrap.cc:492] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P 13dfaa275cba432cb54449dd5f0eac83: Bootstrap starting.
I20260929 19:06:48.804198 28428 tablet_bootstrap.cc:654] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P 13dfaa275cba432cb54449dd5f0eac83: Neither blocks nor log segments found. Creating new log.
I20260929 19:06:48.804615 28428 log.cc:824] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P 13dfaa275cba432cb54449dd5f0eac83: Log is configured to *not* fsync() on all Append() calls
I20260929 19:06:48.804890 28429 tablet_bootstrap.cc:492] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P a0346a7a77c740eeb8b7a55d0b838a2f: Bootstrap starting.
I20260929 19:06:48.805181 28431 tablet_bootstrap.cc:492] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P ea4f0422c2f5493b8eab7e3ef6686aa6: Bootstrap starting.
I20260929 19:06:48.805532 28429 tablet_bootstrap.cc:654] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P a0346a7a77c740eeb8b7a55d0b838a2f: Neither blocks nor log segments found. Creating new log.
I20260929 19:06:48.805797 28431 tablet_bootstrap.cc:654] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P ea4f0422c2f5493b8eab7e3ef6686aa6: Neither blocks nor log segments found. Creating new log.
I20260929 19:06:48.806103 28429 log.cc:824] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P a0346a7a77c740eeb8b7a55d0b838a2f: Log is configured to *not* fsync() on all Append() calls
I20260929 19:06:48.806457 28431 log.cc:824] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P ea4f0422c2f5493b8eab7e3ef6686aa6: Log is configured to *not* fsync() on all Append() calls
I20260929 19:06:48.806846 28429 tablet_bootstrap.cc:492] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P a0346a7a77c740eeb8b7a55d0b838a2f: No bootstrap required, opened a new log
I20260929 19:06:48.806850 28428 tablet_bootstrap.cc:492] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P 13dfaa275cba432cb54449dd5f0eac83: No bootstrap required, opened a new log
I20260929 19:06:48.806953 28429 ts_tablet_manager.cc:1397] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P a0346a7a77c740eeb8b7a55d0b838a2f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260929 19:06:48.806982 28428 ts_tablet_manager.cc:1397] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P 13dfaa275cba432cb54449dd5f0eac83: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 19:06:48.808238 28429 raft_consensus.cc:348] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P a0346a7a77c740eeb8b7a55d0b838a2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13dfaa275cba432cb54449dd5f0eac83" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 40473 } } peers { permanent_uuid: "a0346a7a77c740eeb8b7a55d0b838a2f" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 36237 } } peers { permanent_uuid: "ea4f0422c2f5493b8eab7e3ef6686aa6" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46783 } }
I20260929 19:06:48.808368 28429 raft_consensus.cc:374] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P a0346a7a77c740eeb8b7a55d0b838a2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:06:48.808393 28429 raft_consensus.cc:729] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P a0346a7a77c740eeb8b7a55d0b838a2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0346a7a77c740eeb8b7a55d0b838a2f, State: Initialized, Role: FOLLOWER
I20260929 19:06:48.808274 28428 raft_consensus.cc:348] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P 13dfaa275cba432cb54449dd5f0eac83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13dfaa275cba432cb54449dd5f0eac83" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 40473 } } peers { permanent_uuid: "a0346a7a77c740eeb8b7a55d0b838a2f" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 36237 } } peers { permanent_uuid: "ea4f0422c2f5493b8eab7e3ef6686aa6" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46783 } }
I20260929 19:06:48.808437 28428 raft_consensus.cc:374] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P 13dfaa275cba432cb54449dd5f0eac83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:06:48.808483 28428 raft_consensus.cc:729] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P 13dfaa275cba432cb54449dd5f0eac83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13dfaa275cba432cb54449dd5f0eac83, State: Initialized, Role: FOLLOWER
I20260929 19:06:48.808544 28429 consensus_queue.cc:260] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P a0346a7a77c740eeb8b7a55d0b838a2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13dfaa275cba432cb54449dd5f0eac83" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 40473 } } peers { permanent_uuid: "a0346a7a77c740eeb8b7a55d0b838a2f" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 36237 } } peers { permanent_uuid: "ea4f0422c2f5493b8eab7e3ef6686aa6" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46783 } }
I20260929 19:06:48.808588 28428 consensus_queue.cc:260] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P 13dfaa275cba432cb54449dd5f0eac83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13dfaa275cba432cb54449dd5f0eac83" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 40473 } } peers { permanent_uuid: "a0346a7a77c740eeb8b7a55d0b838a2f" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 36237 } } peers { permanent_uuid: "ea4f0422c2f5493b8eab7e3ef6686aa6" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46783 } }
I20260929 19:06:48.808833 28414 heartbeater.cc:503] Master 127.8.158.126:42669 was elected leader, sending a full tablet report...
I20260929 19:06:48.808861 28428 ts_tablet_manager.cc:1428] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P 13dfaa275cba432cb54449dd5f0eac83: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:06:48.808843 28429 ts_tablet_manager.cc:1428] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P a0346a7a77c740eeb8b7a55d0b838a2f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 19:06:48.808983 28146 heartbeater.cc:503] Master 127.8.158.126:42669 was elected leader, sending a full tablet report...
I20260929 19:06:48.809691 28431 tablet_bootstrap.cc:492] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P ea4f0422c2f5493b8eab7e3ef6686aa6: No bootstrap required, opened a new log
I20260929 19:06:48.809756 28431 ts_tablet_manager.cc:1397] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P ea4f0422c2f5493b8eab7e3ef6686aa6: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260929 19:06:48.811265 28431 raft_consensus.cc:348] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P ea4f0422c2f5493b8eab7e3ef6686aa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13dfaa275cba432cb54449dd5f0eac83" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 40473 } } peers { permanent_uuid: "a0346a7a77c740eeb8b7a55d0b838a2f" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 36237 } } peers { permanent_uuid: "ea4f0422c2f5493b8eab7e3ef6686aa6" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46783 } }
I20260929 19:06:48.811414 28431 raft_consensus.cc:374] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P ea4f0422c2f5493b8eab7e3ef6686aa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:06:48.811460 28431 raft_consensus.cc:729] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P ea4f0422c2f5493b8eab7e3ef6686aa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea4f0422c2f5493b8eab7e3ef6686aa6, State: Initialized, Role: FOLLOWER
I20260929 19:06:48.811566 28431 consensus_queue.cc:260] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P ea4f0422c2f5493b8eab7e3ef6686aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13dfaa275cba432cb54449dd5f0eac83" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 40473 } } peers { permanent_uuid: "a0346a7a77c740eeb8b7a55d0b838a2f" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 36237 } } peers { permanent_uuid: "ea4f0422c2f5493b8eab7e3ef6686aa6" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46783 } }
I20260929 19:06:48.811841 28431 ts_tablet_manager.cc:1428] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P ea4f0422c2f5493b8eab7e3ef6686aa6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:06:48.811841 28280 heartbeater.cc:503] Master 127.8.158.126:42669 was elected leader, sending a full tablet report...
I20260929 19:06:48.849920 28436 raft_consensus.cc:482] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P ea4f0422c2f5493b8eab7e3ef6686aa6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:06:48.850020 28436 raft_consensus.cc:504] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P ea4f0422c2f5493b8eab7e3ef6686aa6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13dfaa275cba432cb54449dd5f0eac83" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 40473 } } peers { permanent_uuid: "a0346a7a77c740eeb8b7a55d0b838a2f" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 36237 } } peers { permanent_uuid: "ea4f0422c2f5493b8eab7e3ef6686aa6" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46783 } }
I20260929 19:06:48.850299 28436 leader_election.cc:288] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P ea4f0422c2f5493b8eab7e3ef6686aa6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13dfaa275cba432cb54449dd5f0eac83 (127.8.158.65:40473), a0346a7a77c740eeb8b7a55d0b838a2f (127.8.158.67:36237)
I20260929 19:06:48.854369 28425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:48.850735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:34371 (local address 127.8.158.65:40473)
0929 19:06:48.850783 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:48.850785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:48.850796 (+ 11us) server_negotiation.cc:407] Connection header received
0929 19:06:48.850824 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:48.850826 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:48.850843 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:48.850928 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:06:48.851123 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:48.851860 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:48.853388 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:48.853982 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:48.854056 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:48.854063 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:48.854246 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 19:06:48.854285 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 19:06:48.854382 28427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:48.850747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:40085 (local address 127.8.158.67:36237)
0929 19:06:48.850809 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:48.850811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:48.850904 (+ 93us) server_negotiation.cc:407] Connection header received
0929 19:06:48.850930 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:48.850931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:48.850939 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:48.850983 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:06:48.851123 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:48.851864 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:48.853388 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:48.853972 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:48.854055 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:48.854063 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:48.854260 (+ 197us) server_negotiation.cc:299] Negotiation successful
0929 19:06:48.854295 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 19:06:48.854426 28282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:48.850532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:40473 (local address 127.8.158.66:34371)
0929 19:06:48.850590 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:48.850603 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:48.850646 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:48.850961 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:48.850962 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:48.850963 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:06:48.851039 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:48.851040 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:48.851956 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:48.851957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:48.853273 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:48.853276 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:48.854069 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:48.854070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:48.854159 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:48.854165 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 19:06:48.854227 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 19:06:48.854273 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":92}
I20260929 19:06:48.854720 28437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:48.850658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.67:36237 (local address 127.8.158.66:40085)
0929 19:06:48.850821 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:48.850829 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:48.850872 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:48.850994 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:48.850994 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:48.850995 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:06:48.851055 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:48.851056 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:48.851956 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:48.851957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:48.853273 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:48.853276 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:48.854069 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:48.854070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:48.854159 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:48.854166 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 19:06:48.854227 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 19:06:48.854272 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":69,"thread_start_us":83,"threads_started":1}
I20260929 19:06:48.854789 28100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ee3b0e82bbe403eb3b9c8e4c5de24ba" candidate_uuid: "ea4f0422c2f5493b8eab7e3ef6686aa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13dfaa275cba432cb54449dd5f0eac83" is_pre_election: true
I20260929 19:06:48.854966 28100 raft_consensus.cc:2454] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P 13dfaa275cba432cb54449dd5f0eac83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea4f0422c2f5493b8eab7e3ef6686aa6 in term 0.
I20260929 19:06:48.855060 28368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ee3b0e82bbe403eb3b9c8e4c5de24ba" candidate_uuid: "ea4f0422c2f5493b8eab7e3ef6686aa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0346a7a77c740eeb8b7a55d0b838a2f" is_pre_election: true
I20260929 19:06:48.855178 28368 raft_consensus.cc:2454] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P a0346a7a77c740eeb8b7a55d0b838a2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea4f0422c2f5493b8eab7e3ef6686aa6 in term 0.
I20260929 19:06:48.855377 28167 leader_election.cc:302] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P ea4f0422c2f5493b8eab7e3ef6686aa6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13dfaa275cba432cb54449dd5f0eac83, ea4f0422c2f5493b8eab7e3ef6686aa6; no voters:
I20260929 19:06:48.855540 28436 raft_consensus.cc:2781] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P ea4f0422c2f5493b8eab7e3ef6686aa6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:06:48.855612 28436 raft_consensus.cc:482] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P ea4f0422c2f5493b8eab7e3ef6686aa6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:06:48.855629 28436 raft_consensus.cc:3037] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P ea4f0422c2f5493b8eab7e3ef6686aa6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:06:48.856341 28436 raft_consensus.cc:504] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P ea4f0422c2f5493b8eab7e3ef6686aa6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13dfaa275cba432cb54449dd5f0eac83" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 40473 } } peers { permanent_uuid: "a0346a7a77c740eeb8b7a55d0b838a2f" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 36237 } } peers { permanent_uuid: "ea4f0422c2f5493b8eab7e3ef6686aa6" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46783 } }
I20260929 19:06:48.856467 28436 leader_election.cc:288] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P ea4f0422c2f5493b8eab7e3ef6686aa6 [CANDIDATE]: Term 1 election: Requested vote from peers 13dfaa275cba432cb54449dd5f0eac83 (127.8.158.65:40473), a0346a7a77c740eeb8b7a55d0b838a2f (127.8.158.67:36237)
I20260929 19:06:48.856643 28368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ee3b0e82bbe403eb3b9c8e4c5de24ba" candidate_uuid: "ea4f0422c2f5493b8eab7e3ef6686aa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0346a7a77c740eeb8b7a55d0b838a2f"
I20260929 19:06:48.856662 28100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ee3b0e82bbe403eb3b9c8e4c5de24ba" candidate_uuid: "ea4f0422c2f5493b8eab7e3ef6686aa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13dfaa275cba432cb54449dd5f0eac83"
I20260929 19:06:48.856716 28368 raft_consensus.cc:3037] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P a0346a7a77c740eeb8b7a55d0b838a2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:06:48.856757 28100 raft_consensus.cc:3037] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P 13dfaa275cba432cb54449dd5f0eac83 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:06:48.857383 28368 raft_consensus.cc:2454] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P a0346a7a77c740eeb8b7a55d0b838a2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea4f0422c2f5493b8eab7e3ef6686aa6 in term 1.
I20260929 19:06:48.857537 28167 leader_election.cc:302] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P ea4f0422c2f5493b8eab7e3ef6686aa6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0346a7a77c740eeb8b7a55d0b838a2f, ea4f0422c2f5493b8eab7e3ef6686aa6; no voters:
I20260929 19:06:48.857549 28100 raft_consensus.cc:2454] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P 13dfaa275cba432cb54449dd5f0eac83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea4f0422c2f5493b8eab7e3ef6686aa6 in term 1.
I20260929 19:06:48.857648 28436 raft_consensus.cc:2781] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P ea4f0422c2f5493b8eab7e3ef6686aa6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:06:48.857800 28436 raft_consensus.cc:686] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P ea4f0422c2f5493b8eab7e3ef6686aa6 [term 1 LEADER]: Becoming Leader. State: Replica: ea4f0422c2f5493b8eab7e3ef6686aa6, State: Running, Role: LEADER
I20260929 19:06:48.857898 28436 consensus_queue.cc:237] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P ea4f0422c2f5493b8eab7e3ef6686aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13dfaa275cba432cb54449dd5f0eac83" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 40473 } } peers { permanent_uuid: "a0346a7a77c740eeb8b7a55d0b838a2f" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 36237 } } peers { permanent_uuid: "ea4f0422c2f5493b8eab7e3ef6686aa6" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46783 } }
I20260929 19:06:48.858618 27955 catalog_manager.cc:5495] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P ea4f0422c2f5493b8eab7e3ef6686aa6 reported cstate change: term changed from 0 to 1, leader changed from <none> to ea4f0422c2f5493b8eab7e3ef6686aa6 (127.8.158.66). New cstate: current_term: 1 leader_uuid: "ea4f0422c2f5493b8eab7e3ef6686aa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13dfaa275cba432cb54449dd5f0eac83" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 40473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0346a7a77c740eeb8b7a55d0b838a2f" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 36237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea4f0422c2f5493b8eab7e3ef6686aa6" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 46783 } health_report { overall_health: HEALTHY } } }
W20260929 19:06:48.865177 28281 tablet.cc:2369] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P ea4f0422c2f5493b8eab7e3ef6686aa6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:06:48.873471 28426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:48.871421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35324 (local address 127.8.158.66:46783)
0929 19:06:48.871475 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:48.871476 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:48.871485 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:06:48.871515 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:48.871516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:48.871530 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:48.871592 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:06:48.871673 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:48.872131 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:48.872812 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:48.872987 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:48.873073 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:48.873187 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:06:48.873267 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:06:48.873319 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:48.873352 (+ 33us) server_negotiation.cc:299] Negotiation successful
0929 19:06:48.873367 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 19:06:48.875181 28368 raft_consensus.cc:1261] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P a0346a7a77c740eeb8b7a55d0b838a2f [term 1 FOLLOWER]: Refusing update from remote peer ea4f0422c2f5493b8eab7e3ef6686aa6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:06:48.875211 28100 raft_consensus.cc:1261] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P 13dfaa275cba432cb54449dd5f0eac83 [term 1 FOLLOWER]: Refusing update from remote peer ea4f0422c2f5493b8eab7e3ef6686aa6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:06:48.875547 28436 consensus_queue.cc:1035] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P ea4f0422c2f5493b8eab7e3ef6686aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0346a7a77c740eeb8b7a55d0b838a2f" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:06:48.875667 28438 consensus_queue.cc:1035] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P ea4f0422c2f5493b8eab7e3ef6686aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13dfaa275cba432cb54449dd5f0eac83" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:06:48.877885 28441 mvcc.cc:205] Tried to move back new op lower bound from 7334743281150222336 to 7334743281082630144. Current Snapshot: MvccSnapshot[applied={T|T < 7334743281150222336}]
I20260929 19:06:48.878476 28442 mvcc.cc:205] Tried to move back new op lower bound from 7334743281150222336 to 7334743281082630144. Current Snapshot: MvccSnapshot[applied={T|T < 7334743281150222336}]
I20260929 19:06:48.880435 27955 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:06:48.880533 27955 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:06:48.881237 27955 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a82b2135e94f49d8afa36ef69599b825: Sending DeleteTablet for 3 replicas of tablet 5ee3b0e82bbe403eb3b9c8e4c5de24ba
I20260929 19:06:48.881588 28348 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ee3b0e82bbe403eb3b9c8e4c5de24ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:06:48 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:44072
I20260929 19:06:48.881596 28080 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ee3b0e82bbe403eb3b9c8e4c5de24ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:06:48 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:41528
I20260929 19:06:48.881821 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 28017
I20260929 19:06:48.881901 28214 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ee3b0e82bbe403eb3b9c8e4c5de24ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:06:48 UTC) from {username='kudu', principal='kudu/127.8.158.126@KRBTEST.COM'} at 127.0.0.1:35308
I20260929 19:06:48.883951 28453 tablet_replica.cc:330] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P ea4f0422c2f5493b8eab7e3ef6686aa6: stopping tablet replica
I20260929 19:06:48.884059 28453 raft_consensus.cc:2229] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P ea4f0422c2f5493b8eab7e3ef6686aa6 [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:06:48.884192 28453 raft_consensus.cc:2258] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P ea4f0422c2f5493b8eab7e3ef6686aa6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:06:48.884375 28452 tablet_replica.cc:330] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P a0346a7a77c740eeb8b7a55d0b838a2f: stopping tablet replica
I20260929 19:06:48.884477 28452 raft_consensus.cc:2229] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P a0346a7a77c740eeb8b7a55d0b838a2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:06:48.884578 28452 raft_consensus.cc:2258] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P a0346a7a77c740eeb8b7a55d0b838a2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:06:48.884642 28453 ts_tablet_manager.cc:1905] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P ea4f0422c2f5493b8eab7e3ef6686aa6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:06:48.884992 28452 ts_tablet_manager.cc:1905] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P a0346a7a77c740eeb8b7a55d0b838a2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:06:48.886338 28453 ts_tablet_manager.cc:1918] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P ea4f0422c2f5493b8eab7e3ef6686aa6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:06:48.886415 28453 log.cc:1196] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P ea4f0422c2f5493b8eab7e3ef6686aa6: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-1/wal/wals/5ee3b0e82bbe403eb3b9c8e4c5de24ba
I20260929 19:06:48.886623 28452 ts_tablet_manager.cc:1918] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P a0346a7a77c740eeb8b7a55d0b838a2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:06:48.886694 28452 log.cc:1196] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P a0346a7a77c740eeb8b7a55d0b838a2f: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708573736807-8825-0/minicluster-data/ts-2/wal/wals/5ee3b0e82bbe403eb3b9c8e4c5de24ba
I20260929 19:06:48.886694 28453 ts_tablet_manager.cc:1939] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P ea4f0422c2f5493b8eab7e3ef6686aa6: Deleting consensus metadata
I20260929 19:06:48.886973 28452 ts_tablet_manager.cc:1939] T 5ee3b0e82bbe403eb3b9c8e4c5de24ba P a0346a7a77c740eeb8b7a55d0b838a2f: Deleting consensus metadata
I20260929 19:06:48.887212 27940 catalog_manager.cc:4841] TS ea4f0422c2f5493b8eab7e3ef6686aa6 (127.8.158.66:46783): tablet 5ee3b0e82bbe403eb3b9c8e4c5de24ba (table test-table [id=8aaf16b1bb6743baa610b7fc84b5120f]) successfully deleted
I20260929 19:06:48.887295 27939 catalog_manager.cc:4841] TS a0346a7a77c740eeb8b7a55d0b838a2f (127.8.158.67:36237): tablet 5ee3b0e82bbe403eb3b9c8e4c5de24ba (table test-table [id=8aaf16b1bb6743baa610b7fc84b5120f]) successfully deleted
W20260929 19:06:48.894251 27939 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.158.65:40473 (error 108)
I20260929 19:06:48.894445 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 28151
I20260929 19:06:48.895339 28424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:48.895205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:40473 (local address 127.0.0.1:41532)
0929 19:06:48.895264 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:48.895300 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.65:40473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 19:06:48.895529 27939 catalog_manager.cc:4584] TS 13dfaa275cba432cb54449dd5f0eac83 (127.8.158.65:40473): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5ee3b0e82bbe403eb3b9c8e4c5de24ba: Network error: Client connection negotiation failed: client connection to 127.8.158.65:40473: connect: Connection refused (error 111)
I20260929 19:06:48.898725 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 28285
I20260929 19:06:48.903581 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 27923
2026-09-29T19:06:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1898 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T19:06:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:06:48Z Disabled control of system clock
I20260929 19:06:48.930298 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:42531
--webserver_interface=127.8.158.126
--webserver_port=0
--builtin_ntp_servers=127.8.158.84:37875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:42531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 19:06:49.008121 28458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:06:49.008313 28458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:06:49.008333 28458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:06:49.009462 28458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:06:49.009502 28458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:06:49.009516 28458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:06:49.009528 28458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:06:49.009577 28458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:06:49.010938 28458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:37875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:42531
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:42531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.158.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:06:49.011173 28458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:06:49.011461 28458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:06:49.016196 28464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:49.016234 28466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:49.016204 28463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:49.016311 28458 server_base.cc:1034] running on GCE node
I20260929 19:06:49.016546 28458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:06:49.016824 28458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:06:49.017985 28458 hybrid_clock.cc:648] HybridClock initialized: now 1790708809017966 us; error 31 us; skew 500 ppm
I20260929 19:06:49.019207 28458 webserver.cc:460] Webserver started at http://127.8.158.126:43391/ using document root <none> and password file <none>
I20260929 19:06:49.019461 28458 fs_manager.cc:359] Metadata directory not provided
I20260929 19:06:49.019531 28458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:06:49.019675 28458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:06:49.020649 28458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "cad2eda191d34576922224bf3cc3b6d2"
format_stamp: "Formatted at 2026-09-29 19:06:49 on dist-test-slave-nmsz"
server_key: "8cadecfdac8675c14b166274e4c25d4a"
server_key_iv: "6643cee298e503d59fc7af542a197cfb"
server_key_version: "encryptionkey@0"
I20260929 19:06:49.020980 28458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "cad2eda191d34576922224bf3cc3b6d2"
format_stamp: "Formatted at 2026-09-29 19:06:49 on dist-test-slave-nmsz"
server_key: "8cadecfdac8675c14b166274e4c25d4a"
server_key_iv: "6643cee298e503d59fc7af542a197cfb"
server_key_version: "encryptionkey@0"
I20260929 19:06:49.022254 28458 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:06:49.023029 28472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:49.023192 28458 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:06:49.023296 28458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "cad2eda191d34576922224bf3cc3b6d2"
format_stamp: "Formatted at 2026-09-29 19:06:49 on dist-test-slave-nmsz"
server_key: "8cadecfdac8675c14b166274e4c25d4a"
server_key_iv: "6643cee298e503d59fc7af542a197cfb"
server_key_version: "encryptionkey@0"
I20260929 19:06:49.023387 28458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:06:49.040176 28458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:06:49.040458 28458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:06:49.040607 28458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:06:49.044376 28458 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.126:42531
I20260929 19:06:49.044404 28524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.126:42531 every 8 connection(s)
I20260929 19:06:49.044704 28458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:06:49.045883 28525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:06:49.048508 28525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cad2eda191d34576922224bf3cc3b6d2: Bootstrap starting.
I20260929 19:06:49.048997 28525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cad2eda191d34576922224bf3cc3b6d2: Neither blocks nor log segments found. Creating new log.
I20260929 19:06:49.049230 28525 log.cc:824] T 00000000000000000000000000000000 P cad2eda191d34576922224bf3cc3b6d2: Log is configured to *not* fsync() on all Append() calls
I20260929 19:06:49.049821 28525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cad2eda191d34576922224bf3cc3b6d2: No bootstrap required, opened a new log
I20260929 19:06:49.051388 28525 raft_consensus.cc:348] T 00000000000000000000000000000000 P cad2eda191d34576922224bf3cc3b6d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cad2eda191d34576922224bf3cc3b6d2" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 42531 } }
I20260929 19:06:49.051512 28525 raft_consensus.cc:374] T 00000000000000000000000000000000 P cad2eda191d34576922224bf3cc3b6d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:06:49.051533 28525 raft_consensus.cc:729] T 00000000000000000000000000000000 P cad2eda191d34576922224bf3cc3b6d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cad2eda191d34576922224bf3cc3b6d2, State: Initialized, Role: FOLLOWER
I20260929 19:06:49.051656 28525 consensus_queue.cc:260] T 00000000000000000000000000000000 P cad2eda191d34576922224bf3cc3b6d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cad2eda191d34576922224bf3cc3b6d2" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 42531 } }
I20260929 19:06:49.051735 28525 raft_consensus.cc:388] T 00000000000000000000000000000000 P cad2eda191d34576922224bf3cc3b6d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:06:49.051757 28525 raft_consensus.cc:482] T 00000000000000000000000000000000 P cad2eda191d34576922224bf3cc3b6d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:06:49.051783 28525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cad2eda191d34576922224bf3cc3b6d2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:06:49.052407 28525 raft_consensus.cc:504] T 00000000000000000000000000000000 P cad2eda191d34576922224bf3cc3b6d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cad2eda191d34576922224bf3cc3b6d2" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 42531 } }
I20260929 19:06:49.052516 28525 leader_election.cc:302] T 00000000000000000000000000000000 P cad2eda191d34576922224bf3cc3b6d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cad2eda191d34576922224bf3cc3b6d2; no voters:
I20260929 19:06:49.052670 28525 leader_election.cc:288] T 00000000000000000000000000000000 P cad2eda191d34576922224bf3cc3b6d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:06:49.052794 28528 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cad2eda191d34576922224bf3cc3b6d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:06:49.052896 28525 sys_catalog.cc:564] T 00000000000000000000000000000000 P cad2eda191d34576922224bf3cc3b6d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:06:49.052960 28528 raft_consensus.cc:686] T 00000000000000000000000000000000 P cad2eda191d34576922224bf3cc3b6d2 [term 1 LEADER]: Becoming Leader. State: Replica: cad2eda191d34576922224bf3cc3b6d2, State: Running, Role: LEADER
I20260929 19:06:49.053109 28528 consensus_queue.cc:237] T 00000000000000000000000000000000 P cad2eda191d34576922224bf3cc3b6d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cad2eda191d34576922224bf3cc3b6d2" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 42531 } }
I20260929 19:06:49.053627 28529 sys_catalog.cc:455] T 00000000000000000000000000000000 P cad2eda191d34576922224bf3cc3b6d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cad2eda191d34576922224bf3cc3b6d2. Latest consensus state: current_term: 1 leader_uuid: "cad2eda191d34576922224bf3cc3b6d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cad2eda191d34576922224bf3cc3b6d2" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 42531 } } }
I20260929 19:06:49.053705 28529 sys_catalog.cc:458] T 00000000000000000000000000000000 P cad2eda191d34576922224bf3cc3b6d2 [sys.catalog]: This master's current role is: LEADER
I20260929 19:06:49.053670 28530 sys_catalog.cc:455] T 00000000000000000000000000000000 P cad2eda191d34576922224bf3cc3b6d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cad2eda191d34576922224bf3cc3b6d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cad2eda191d34576922224bf3cc3b6d2" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 42531 } } }
I20260929 19:06:49.053805 28530 sys_catalog.cc:458] T 00000000000000000000000000000000 P cad2eda191d34576922224bf3cc3b6d2 [sys.catalog]: This master's current role is: LEADER
I20260929 19:06:49.054191 28537 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:06:49.054400 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 28458
I20260929 19:06:49.054494 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:06:49.054670 8825 external_mini_cluster.cc:1428] Setting key a687c6d786ac5feb613c485ecee87760
I20260929 19:06:49.055215 28537 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:06:49.056702 28537 catalog_manager.cc:1347] Generated new cluster ID: 9971de51d9034e3aaf2ada0f1bd18cca
I20260929 19:06:49.056756 28537 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:06:49.063704 28546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:49.055514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34772 (local address 127.8.158.126:42531)
0929 19:06:49.055705 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:49.055714 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:49.058568 (+ 2854us) server_negotiation.cc:407] Connection header received
0929 19:06:49.058714 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:49.058720 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:49.058783 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:49.058917 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:49.059136 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:49.060502 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:49.061626 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:49.062213 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:49.062337 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:49.063041 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:49.063045 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:49.063045 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:06:49.063232 (+ 187us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:49.063270 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:06:49.063282 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:49.063343 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:49.063458 (+ 115us) server_negotiation.cc:299] Negotiation successful
0929 19:06:49.063527 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":82,"threads_started":1}
I20260929 19:06:49.068574 28537 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:06:49.069036 28537 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:06:49.074832 28537 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cad2eda191d34576922224bf3cc3b6d2: Generated new TSK 0
I20260929 19:06:49.074992 28537 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:06:49.082227 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.65:0
--local_ip_for_outbound_sockets=127.8.158.65
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:42531
--builtin_ntp_servers=127.8.158.84:37875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 19:06:49.168633 28549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:06:49.168820 28549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:06:49.168841 28549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:06:49.169955 28549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:06:49.169994 28549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:06:49.170018 28549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.65
I20260929 19:06:49.171353 28549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:37875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.158.65
--webserver_port=0
--tserver_master_addrs=127.8.158.126:42531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.65
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:06:49.171568 28549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:06:49.171862 28549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:06:49.177395 28555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:49.177428 28557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:49.177418 28554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:49.177625 28549 server_base.cc:1034] running on GCE node
I20260929 19:06:49.177879 28549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:06:49.178154 28549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:06:49.179309 28549 hybrid_clock.cc:648] HybridClock initialized: now 1790708809179284 us; error 42 us; skew 500 ppm
I20260929 19:06:49.180593 28549 webserver.cc:460] Webserver started at http://127.8.158.65:43501/ using document root <none> and password file <none>
I20260929 19:06:49.180827 28549 fs_manager.cc:359] Metadata directory not provided
I20260929 19:06:49.180871 28549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:06:49.181006 28549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:06:49.182035 28549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-0/data/instance:
uuid: "c387b46178d24994ba4bba382cc3d7a2"
format_stamp: "Formatted at 2026-09-29 19:06:49 on dist-test-slave-nmsz"
server_key: "cdefd3acae0bb46eec6977d64f0d0ae4"
server_key_iv: "2eac0d60672c1839c02eea378ce35d9e"
server_key_version: "encryptionkey@0"
I20260929 19:06:49.182410 28549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance:
uuid: "c387b46178d24994ba4bba382cc3d7a2"
format_stamp: "Formatted at 2026-09-29 19:06:49 on dist-test-slave-nmsz"
server_key: "cdefd3acae0bb46eec6977d64f0d0ae4"
server_key_iv: "2eac0d60672c1839c02eea378ce35d9e"
server_key_version: "encryptionkey@0"
I20260929 19:06:49.183700 28549 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:06:49.184448 28563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:49.184597 28549 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:06:49.184677 28549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-0/wal
uuid: "c387b46178d24994ba4bba382cc3d7a2"
format_stamp: "Formatted at 2026-09-29 19:06:49 on dist-test-slave-nmsz"
server_key: "cdefd3acae0bb46eec6977d64f0d0ae4"
server_key_iv: "2eac0d60672c1839c02eea378ce35d9e"
server_key_version: "encryptionkey@0"
I20260929 19:06:49.184744 28549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:06:49.192860 28549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:06:49.193118 28549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:06:49.193255 28549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:06:49.193457 28549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:06:49.193745 28549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:06:49.193778 28549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:49.193796 28549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:06:49.193841 28549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:49.199662 28549 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.65:42727
I20260929 19:06:49.199697 28676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.65:42727 every 8 connection(s)
I20260929 19:06:49.200003 28549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-0/data/info.pb
I20260929 19:06:49.204906 28546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:49.200713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.65:52441 (local address 127.8.158.126:42531)
0929 19:06:49.200777 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:49.200779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:49.201035 (+ 256us) server_negotiation.cc:407] Connection header received
0929 19:06:49.201510 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:49.201511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:49.201523 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:49.201585 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:49.202197 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:49.202702 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:49.203635 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:49.203821 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:49.203879 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:49.204687 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:49.204688 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:49.204688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:06:49.204693 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:49.204706 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:06:49.204710 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:49.204753 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:49.204818 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 19:06:49.204851 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 19:06:49.205108 28679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:49.200719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:42531 (local address 127.8.158.65:52441)
0929 19:06:49.200889 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:49.200928 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:49.201374 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:49.201635 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:49.201637 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:49.201645 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:06:49.202096 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:49.202099 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:49.202782 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:49.202783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:49.203550 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:49.203552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:49.203895 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:49.203896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:49.204016 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:49.204554 (+ 538us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:06:49.204555 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:06:49.204581 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:06:49.204602 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:06:49.204754 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:06:49.204756 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:06:49.204758 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 19:06:49.204803 (+ 45us) client_negotiation.cc:240] Negotiation successful
0929 19:06:49.204884 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":91,"threads_started":1}
I20260929 19:06:49.205657 28677 heartbeater.cc:347] Connected to a master server at 127.8.158.126:42531
I20260929 19:06:49.205729 28677 heartbeater.cc:464] Registering TS with master...
I20260929 19:06:49.205914 28677 heartbeater.cc:511] Master 127.8.158.126:42531 requested a full tablet report, sending...
I20260929 19:06:49.206481 28489 ts_manager.cc:194] Registered new tserver with Master: c387b46178d24994ba4bba382cc3d7a2 (127.8.158.65:42727)
I20260929 19:06:49.207123 28489 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.158.65:52441
I20260929 19:06:49.207204 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 28549
I20260929 19:06:49.207322 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-0/wal/instance
I20260929 19:06:49.207512 8825 external_mini_cluster.cc:1428] Setting key e7c5f98684219e44c6435dfc652720ce
I20260929 19:06:49.208360 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.66:0
--local_ip_for_outbound_sockets=127.8.158.66
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:42531
--builtin_ntp_servers=127.8.158.84:37875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 19:06:49.285676 28680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:06:49.285858 28680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:06:49.285879 28680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:06:49.286955 28680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:06:49.287001 28680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:06:49.287029 28680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.66
I20260929 19:06:49.288427 28680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:37875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.158.66
--webserver_port=0
--tserver_master_addrs=127.8.158.126:42531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.66
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:06:49.288589 28680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:06:49.288793 28680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:06:49.293617 28685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:49.293722 28680 server_base.cc:1034] running on GCE node
W20260929 19:06:49.293591 28686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:49.293609 28688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:49.294037 28680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:06:49.294272 28680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:06:49.295450 28680 hybrid_clock.cc:648] HybridClock initialized: now 1790708809295440 us; error 27 us; skew 500 ppm
I20260929 19:06:49.296597 28680 webserver.cc:460] Webserver started at http://127.8.158.66:34663/ using document root <none> and password file <none>
I20260929 19:06:49.296824 28680 fs_manager.cc:359] Metadata directory not provided
I20260929 19:06:49.296892 28680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:06:49.297063 28680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:06:49.298053 28680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-1/data/instance:
uuid: "1bd37eb0b27040b79130291494df1e50"
format_stamp: "Formatted at 2026-09-29 19:06:49 on dist-test-slave-nmsz"
server_key: "80548fa43fc9e40c78691d6638a04d43"
server_key_iv: "f38bdc74269e73415352065472537e89"
server_key_version: "encryptionkey@0"
I20260929 19:06:49.298434 28680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance:
uuid: "1bd37eb0b27040b79130291494df1e50"
format_stamp: "Formatted at 2026-09-29 19:06:49 on dist-test-slave-nmsz"
server_key: "80548fa43fc9e40c78691d6638a04d43"
server_key_iv: "f38bdc74269e73415352065472537e89"
server_key_version: "encryptionkey@0"
I20260929 19:06:49.299690 28680 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:06:49.300433 28694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:49.300633 28680 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:06:49.300729 28680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-1/wal
uuid: "1bd37eb0b27040b79130291494df1e50"
format_stamp: "Formatted at 2026-09-29 19:06:49 on dist-test-slave-nmsz"
server_key: "80548fa43fc9e40c78691d6638a04d43"
server_key_iv: "f38bdc74269e73415352065472537e89"
server_key_version: "encryptionkey@0"
I20260929 19:06:49.300813 28680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:06:49.311708 28680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:06:49.311962 28680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:06:49.312108 28680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:06:49.312312 28680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:06:49.312611 28680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:06:49.312664 28680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:49.312705 28680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:06:49.312731 28680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:49.318303 28680 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.66:35165
I20260929 19:06:49.318370 28807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.66:35165 every 8 connection(s)
I20260929 19:06:49.318662 28680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-1/data/info.pb
I20260929 19:06:49.322592 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 28680
I20260929 19:06:49.322696 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-1/wal/instance
I20260929 19:06:49.322882 8825 external_mini_cluster.cc:1428] Setting key aa7ea58e15e3ce265243374c128a6769
I20260929 19:06:49.323841 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.158.67:0
--local_ip_for_outbound_sockets=127.8.158.67
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:42531
--builtin_ntp_servers=127.8.158.84:37875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 19:06:49.323889 28546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:49.319315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.66:41513 (local address 127.8.158.126:42531)
0929 19:06:49.319383 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:49.319385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:49.319605 (+ 220us) server_negotiation.cc:407] Connection header received
0929 19:06:49.320105 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:49.320106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:49.320120 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:49.320171 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:49.320824 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:49.321322 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:49.322254 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:49.322561 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:49.322639 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:49.323616 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:49.323618 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:49.323618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:06:49.323623 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:49.323637 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:06:49.323640 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:49.323692 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:49.323784 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 19:06:49.323813 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 19:06:49.324095 28810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:49.319309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:42531 (local address 127.8.158.66:41513)
0929 19:06:49.319489 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:49.319519 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:49.319979 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:49.320237 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:49.320240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:49.320248 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:06:49.320710 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:49.320714 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:49.321417 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:49.321418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:49.322160 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:49.322163 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:49.322644 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:49.322646 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:49.322796 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:49.323467 (+ 671us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:06:49.323468 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:06:49.323504 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:06:49.323531 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:06:49.323694 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:06:49.323699 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:06:49.323702 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:06:49.323787 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 19:06:49.323881 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":81,"threads_started":1}
I20260929 19:06:49.324694 28808 heartbeater.cc:347] Connected to a master server at 127.8.158.126:42531
I20260929 19:06:49.324767 28808 heartbeater.cc:464] Registering TS with master...
I20260929 19:06:49.324965 28808 heartbeater.cc:511] Master 127.8.158.126:42531 requested a full tablet report, sending...
I20260929 19:06:49.325380 28489 ts_manager.cc:194] Registered new tserver with Master: 1bd37eb0b27040b79130291494df1e50 (127.8.158.66:35165)
I20260929 19:06:49.325762 28489 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.158.66:41513
W20260929 19:06:49.409080 28811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:06:49.409315 28811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:06:49.409358 28811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:06:49.410624 28811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:06:49.410689 28811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:06:49.410737 28811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.158.67
I20260929 19:06:49.412226 28811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:37875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.158.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.158.67
--webserver_port=0
--tserver_master_addrs=127.8.158.126:42531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.158.67
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:06:49.412501 28811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:06:49.412810 28811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:06:49.418298 28819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:49.418356 28817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:49.418249 28816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:49.418591 28811 server_base.cc:1034] running on GCE node
I20260929 19:06:49.419032 28811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:06:49.419369 28811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:06:49.420562 28811 hybrid_clock.cc:648] HybridClock initialized: now 1790708809420568 us; error 51 us; skew 500 ppm
I20260929 19:06:49.421954 28811 webserver.cc:460] Webserver started at http://127.8.158.67:33373/ using document root <none> and password file <none>
I20260929 19:06:49.422207 28811 fs_manager.cc:359] Metadata directory not provided
I20260929 19:06:49.422281 28811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:06:49.422463 28811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:06:49.423496 28811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-2/data/instance:
uuid: "c76ece8b4c72461281dedc35e7cbd0c3"
format_stamp: "Formatted at 2026-09-29 19:06:49 on dist-test-slave-nmsz"
server_key: "9e1b6c290947b9576a9993cf84298802"
server_key_iv: "b6513ba85beb1e5713d0c024466f3e52"
server_key_version: "encryptionkey@0"
I20260929 19:06:49.423841 28811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance:
uuid: "c76ece8b4c72461281dedc35e7cbd0c3"
format_stamp: "Formatted at 2026-09-29 19:06:49 on dist-test-slave-nmsz"
server_key: "9e1b6c290947b9576a9993cf84298802"
server_key_iv: "b6513ba85beb1e5713d0c024466f3e52"
server_key_version: "encryptionkey@0"
I20260929 19:06:49.425107 28811 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 19:06:49.425977 28825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:49.426187 28811 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:06:49.426288 28811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-2/wal
uuid: "c76ece8b4c72461281dedc35e7cbd0c3"
format_stamp: "Formatted at 2026-09-29 19:06:49 on dist-test-slave-nmsz"
server_key: "9e1b6c290947b9576a9993cf84298802"
server_key_iv: "b6513ba85beb1e5713d0c024466f3e52"
server_key_version: "encryptionkey@0"
I20260929 19:06:49.426385 28811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:06:49.444736 28811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:06:49.445050 28811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:06:49.445204 28811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:06:49.445425 28811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:06:49.445747 28811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:06:49.445801 28811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:49.445837 28811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:06:49.445863 28811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:49.451551 28811 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.67:41719
I20260929 19:06:49.451596 28938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.67:41719 every 8 connection(s)
I20260929 19:06:49.451890 28811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-2/data/info.pb
I20260929 19:06:49.457705 28546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:49.452598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:51425 (local address 127.8.158.126:42531)
0929 19:06:49.452719 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:49.452722 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:49.452956 (+ 234us) server_negotiation.cc:407] Connection header received
0929 19:06:49.453649 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:49.453650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:49.453664 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:49.453725 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:49.454524 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:49.455106 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:49.456200 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:49.456419 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:49.456505 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:49.457382 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:49.457383 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:49.457384 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:06:49.457388 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:49.457405 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:06:49.457414 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:49.457475 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:49.457572 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 19:06:49.457624 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 19:06:49.457890 28941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:49.452579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:42531 (local address 127.8.158.67:51425)
0929 19:06:49.452816 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:49.452852 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:49.453537 (+ 685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:49.453796 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:49.453800 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:49.453810 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:06:49.454412 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:49.454418 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:49.455230 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:49.455231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:49.456082 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:49.456084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:49.456510 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:49.456511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:49.456664 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:49.457206 (+ 542us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:06:49.457207 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:06:49.457235 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:06:49.457258 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:06:49.457476 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:06:49.457482 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:06:49.457485 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:06:49.457575 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 19:06:49.457682 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":88,"threads_started":1}
I20260929 19:06:49.458457 28939 heartbeater.cc:347] Connected to a master server at 127.8.158.126:42531
I20260929 19:06:49.458549 28939 heartbeater.cc:464] Registering TS with master...
I20260929 19:06:49.458720 28939 heartbeater.cc:511] Master 127.8.158.126:42531 requested a full tablet report, sending...
I20260929 19:06:49.459059 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 28811
I20260929 19:06:49.459165 28489 ts_manager.cc:194] Registered new tserver with Master: c76ece8b4c72461281dedc35e7cbd0c3 (127.8.158.67:41719)
I20260929 19:06:49.459183 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-2/wal/instance
I20260929 19:06:49.459390 8825 external_mini_cluster.cc:1428] Setting key b4314603236d937d40b3b9e5ae03a228
I20260929 19:06:49.459569 28489 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.158.67:51425
I20260929 19:06:49.460286 8825 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:06:49.464375 28546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:49.461580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34778 (local address 127.8.158.126:42531)
0929 19:06:49.461646 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:49.461648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:49.461777 (+ 129us) server_negotiation.cc:407] Connection header received
0929 19:06:49.461821 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:49.461822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:49.461832 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:49.461879 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:49.462039 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:49.462553 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:49.463321 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:49.463496 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:49.463551 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:49.464183 (+ 632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:49.464184 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:49.464185 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:06:49.464189 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:49.464200 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:06:49.464203 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:49.464248 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:49.464306 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 19:06:49.464327 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 19:06:49.465838 28489 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34778:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:06:49.467413 28489 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:06:49.473531 28949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:49.469497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.66:35165 (local address 127.0.0.1:51114)
0929 19:06:49.469976 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:49.469985 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:49.470035 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:49.470264 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:49.470265 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:49.470267 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:06:49.470299 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:49.470301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:49.471559 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:49.471560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:49.472364 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:49.472366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:49.473254 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:49.473255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:49.473334 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:49.473339 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 19:06:49.473373 (+ 34us) client_negotiation.cc:240] Negotiation successful
0929 19:06:49.473423 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":88,"threads_started":1}
I20260929 19:06:49.473713 28950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:49.469565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42998 (local address 127.8.158.65:42727)
0929 19:06:49.469890 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:49.469895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:49.469908 (+ 13us) server_negotiation.cc:407] Connection header received
0929 19:06:49.469949 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:49.469951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:49.469967 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:49.470071 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:06:49.470242 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:49.471403 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:49.472667 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:49.473147 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:49.473238 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:49.473325 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:49.473573 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 19:06:49.473619 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":94,"threads_started":1}
I20260929 19:06:49.473711 28951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:49.469706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60196 (local address 127.8.158.67:41719)
0929 19:06:49.469873 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:49.469878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:49.470160 (+ 282us) server_negotiation.cc:407] Connection header received
0929 19:06:49.470218 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:49.470220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:49.470233 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:49.470489 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:06:49.470537 (+ 48us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:49.471209 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:49.472655 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:49.473333 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:49.473401 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:49.473477 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:49.473515 (+ 38us) server_negotiation.cc:299] Negotiation successful
0929 19:06:49.473549 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":79,"threads_started":1}
I20260929 19:06:49.473737 28952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:49.469719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51114 (local address 127.8.158.66:35165)
0929 19:06:49.469913 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:49.469919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:49.470054 (+ 135us) server_negotiation.cc:407] Connection header received
0929 19:06:49.470096 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:49.470098 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:49.470113 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:49.470199 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:06:49.470404 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:49.471460 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:49.472451 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:49.473151 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:49.473507 (+ 356us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:49.473589 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:49.473612 (+ 23us) server_negotiation.cc:299] Negotiation successful
0929 19:06:49.473655 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":93,"threads_started":1}
I20260929 19:06:49.474090 28948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:49.469471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:42727 (local address 127.0.0.1:42998)
0929 19:06:49.469765 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:49.469805 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:49.469872 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:49.470082 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:49.470085 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:49.470088 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:06:49.470154 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:49.470156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:49.471490 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:49.471491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:49.472573 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:49.472574 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:49.473240 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:49.473241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:49.473315 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:49.473328 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 19:06:49.473400 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 19:06:49.473444 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":26,"thread_start_us":64,"threads_started":1}
I20260929 19:06:49.474243 28947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:49.469497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.67:41719 (local address 127.0.0.1:60196)
0929 19:06:49.470106 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:49.470115 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:49.470160 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:49.470375 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:49.470375 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:49.470378 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:06:49.470424 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:49.470426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:49.471300 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:49.471301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:49.472567 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:49.472570 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:49.473403 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:49.473404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:49.473453 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:49.473456 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:06:49.473491 (+ 35us) client_negotiation.cc:240] Negotiation successful
0929 19:06:49.473526 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"mutex_wait_us":13,"thread_start_us":74,"threads_started":1}
I20260929 19:06:49.474385 28742 tablet_service.cc:1467] Processing CreateTablet for tablet a48f92f7058443bca54a4c94f51a4800 (DEFAULT_TABLE table=test-table [id=b840c3885ada42809bdc5f8ed5a6cfa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:06:49.474761 28742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a48f92f7058443bca54a4c94f51a4800. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:06:49.474795 28611 tablet_service.cc:1467] Processing CreateTablet for tablet a48f92f7058443bca54a4c94f51a4800 (DEFAULT_TABLE table=test-table [id=b840c3885ada42809bdc5f8ed5a6cfa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:06:49.474843 28873 tablet_service.cc:1467] Processing CreateTablet for tablet a48f92f7058443bca54a4c94f51a4800 (DEFAULT_TABLE table=test-table [id=b840c3885ada42809bdc5f8ed5a6cfa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:06:49.475121 28611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a48f92f7058443bca54a4c94f51a4800. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:06:49.475121 28873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a48f92f7058443bca54a4c94f51a4800. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:06:49.477255 28953 tablet_bootstrap.cc:492] T a48f92f7058443bca54a4c94f51a4800 P 1bd37eb0b27040b79130291494df1e50: Bootstrap starting.
I20260929 19:06:49.478066 28953 tablet_bootstrap.cc:654] T a48f92f7058443bca54a4c94f51a4800 P 1bd37eb0b27040b79130291494df1e50: Neither blocks nor log segments found. Creating new log.
I20260929 19:06:49.478307 28954 tablet_bootstrap.cc:492] T a48f92f7058443bca54a4c94f51a4800 P c387b46178d24994ba4bba382cc3d7a2: Bootstrap starting.
I20260929 19:06:49.478444 28953 log.cc:824] T a48f92f7058443bca54a4c94f51a4800 P 1bd37eb0b27040b79130291494df1e50: Log is configured to *not* fsync() on all Append() calls
I20260929 19:06:49.478963 28954 tablet_bootstrap.cc:654] T a48f92f7058443bca54a4c94f51a4800 P c387b46178d24994ba4bba382cc3d7a2: Neither blocks nor log segments found. Creating new log.
I20260929 19:06:49.479195 28953 tablet_bootstrap.cc:492] T a48f92f7058443bca54a4c94f51a4800 P 1bd37eb0b27040b79130291494df1e50: No bootstrap required, opened a new log
I20260929 19:06:49.479269 28953 ts_tablet_manager.cc:1397] T a48f92f7058443bca54a4c94f51a4800 P 1bd37eb0b27040b79130291494df1e50: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:06:49.479267 28954 log.cc:824] T a48f92f7058443bca54a4c94f51a4800 P c387b46178d24994ba4bba382cc3d7a2: Log is configured to *not* fsync() on all Append() calls
I20260929 19:06:49.479570 28955 tablet_bootstrap.cc:492] T a48f92f7058443bca54a4c94f51a4800 P c76ece8b4c72461281dedc35e7cbd0c3: Bootstrap starting.
I20260929 19:06:49.479924 28954 tablet_bootstrap.cc:492] T a48f92f7058443bca54a4c94f51a4800 P c387b46178d24994ba4bba382cc3d7a2: No bootstrap required, opened a new log
I20260929 19:06:49.480005 28954 ts_tablet_manager.cc:1397] T a48f92f7058443bca54a4c94f51a4800 P c387b46178d24994ba4bba382cc3d7a2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 19:06:49.480217 28955 tablet_bootstrap.cc:654] T a48f92f7058443bca54a4c94f51a4800 P c76ece8b4c72461281dedc35e7cbd0c3: Neither blocks nor log segments found. Creating new log.
I20260929 19:06:49.480533 28955 log.cc:824] T a48f92f7058443bca54a4c94f51a4800 P c76ece8b4c72461281dedc35e7cbd0c3: Log is configured to *not* fsync() on all Append() calls
I20260929 19:06:49.481194 28955 tablet_bootstrap.cc:492] T a48f92f7058443bca54a4c94f51a4800 P c76ece8b4c72461281dedc35e7cbd0c3: No bootstrap required, opened a new log
I20260929 19:06:49.481261 28955 ts_tablet_manager.cc:1397] T a48f92f7058443bca54a4c94f51a4800 P c76ece8b4c72461281dedc35e7cbd0c3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:06:49.481966 28954 raft_consensus.cc:348] T a48f92f7058443bca54a4c94f51a4800 P c387b46178d24994ba4bba382cc3d7a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd37eb0b27040b79130291494df1e50" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 35165 } } peers { permanent_uuid: "c387b46178d24994ba4bba382cc3d7a2" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 42727 } } peers { permanent_uuid: "c76ece8b4c72461281dedc35e7cbd0c3" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 41719 } }
I20260929 19:06:49.481985 28953 raft_consensus.cc:348] T a48f92f7058443bca54a4c94f51a4800 P 1bd37eb0b27040b79130291494df1e50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd37eb0b27040b79130291494df1e50" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 35165 } } peers { permanent_uuid: "c387b46178d24994ba4bba382cc3d7a2" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 42727 } } peers { permanent_uuid: "c76ece8b4c72461281dedc35e7cbd0c3" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 41719 } }
I20260929 19:06:49.482126 28954 raft_consensus.cc:374] T a48f92f7058443bca54a4c94f51a4800 P c387b46178d24994ba4bba382cc3d7a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:06:49.482165 28953 raft_consensus.cc:374] T a48f92f7058443bca54a4c94f51a4800 P 1bd37eb0b27040b79130291494df1e50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:06:49.482167 28954 raft_consensus.cc:729] T a48f92f7058443bca54a4c94f51a4800 P c387b46178d24994ba4bba382cc3d7a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c387b46178d24994ba4bba382cc3d7a2, State: Initialized, Role: FOLLOWER
I20260929 19:06:49.482198 28953 raft_consensus.cc:729] T a48f92f7058443bca54a4c94f51a4800 P 1bd37eb0b27040b79130291494df1e50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bd37eb0b27040b79130291494df1e50, State: Initialized, Role: FOLLOWER
I20260929 19:06:49.482300 28953 consensus_queue.cc:260] T a48f92f7058443bca54a4c94f51a4800 P 1bd37eb0b27040b79130291494df1e50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd37eb0b27040b79130291494df1e50" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 35165 } } peers { permanent_uuid: "c387b46178d24994ba4bba382cc3d7a2" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 42727 } } peers { permanent_uuid: "c76ece8b4c72461281dedc35e7cbd0c3" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 41719 } }
I20260929 19:06:49.482276 28954 consensus_queue.cc:260] T a48f92f7058443bca54a4c94f51a4800 P c387b46178d24994ba4bba382cc3d7a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd37eb0b27040b79130291494df1e50" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 35165 } } peers { permanent_uuid: "c387b46178d24994ba4bba382cc3d7a2" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 42727 } } peers { permanent_uuid: "c76ece8b4c72461281dedc35e7cbd0c3" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 41719 } }
I20260929 19:06:49.482584 28953 ts_tablet_manager.cc:1428] T a48f92f7058443bca54a4c94f51a4800 P 1bd37eb0b27040b79130291494df1e50: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260929 19:06:49.482610 28954 ts_tablet_manager.cc:1428] T a48f92f7058443bca54a4c94f51a4800 P c387b46178d24994ba4bba382cc3d7a2: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260929 19:06:49.482738 28677 heartbeater.cc:503] Master 127.8.158.126:42531 was elected leader, sending a full tablet report...
I20260929 19:06:49.483628 28955 raft_consensus.cc:348] T a48f92f7058443bca54a4c94f51a4800 P c76ece8b4c72461281dedc35e7cbd0c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd37eb0b27040b79130291494df1e50" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 35165 } } peers { permanent_uuid: "c387b46178d24994ba4bba382cc3d7a2" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 42727 } } peers { permanent_uuid: "c76ece8b4c72461281dedc35e7cbd0c3" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 41719 } }
I20260929 19:06:49.483777 28955 raft_consensus.cc:374] T a48f92f7058443bca54a4c94f51a4800 P c76ece8b4c72461281dedc35e7cbd0c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:06:49.483829 28955 raft_consensus.cc:729] T a48f92f7058443bca54a4c94f51a4800 P c76ece8b4c72461281dedc35e7cbd0c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c76ece8b4c72461281dedc35e7cbd0c3, State: Initialized, Role: FOLLOWER
I20260929 19:06:49.483932 28955 consensus_queue.cc:260] T a48f92f7058443bca54a4c94f51a4800 P c76ece8b4c72461281dedc35e7cbd0c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd37eb0b27040b79130291494df1e50" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 35165 } } peers { permanent_uuid: "c387b46178d24994ba4bba382cc3d7a2" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 42727 } } peers { permanent_uuid: "c76ece8b4c72461281dedc35e7cbd0c3" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 41719 } }
I20260929 19:06:49.484167 28955 ts_tablet_manager.cc:1428] T a48f92f7058443bca54a4c94f51a4800 P c76ece8b4c72461281dedc35e7cbd0c3: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260929 19:06:49.484195 28939 heartbeater.cc:503] Master 127.8.158.126:42531 was elected leader, sending a full tablet report...
I20260929 19:06:49.484268 28808 heartbeater.cc:503] Master 127.8.158.126:42531 was elected leader, sending a full tablet report...
I20260929 19:06:49.495733 28961 raft_consensus.cc:482] T a48f92f7058443bca54a4c94f51a4800 P c76ece8b4c72461281dedc35e7cbd0c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:06:49.495829 28961 raft_consensus.cc:504] T a48f92f7058443bca54a4c94f51a4800 P c76ece8b4c72461281dedc35e7cbd0c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd37eb0b27040b79130291494df1e50" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 35165 } } peers { permanent_uuid: "c387b46178d24994ba4bba382cc3d7a2" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 42727 } } peers { permanent_uuid: "c76ece8b4c72461281dedc35e7cbd0c3" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 41719 } }
I20260929 19:06:49.496181 28961 leader_election.cc:288] T a48f92f7058443bca54a4c94f51a4800 P c76ece8b4c72461281dedc35e7cbd0c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1bd37eb0b27040b79130291494df1e50 (127.8.158.66:35165), c387b46178d24994ba4bba382cc3d7a2 (127.8.158.65:42727)
I20260929 19:06:49.499920 28950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:49.496419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:34227 (local address 127.8.158.65:42727)
0929 19:06:49.496479 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:49.496481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:49.496572 (+ 91us) server_negotiation.cc:407] Connection header received
0929 19:06:49.496673 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:49.496674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:49.496684 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:49.496713 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:06:49.496897 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:49.497585 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:49.499037 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:49.499582 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:49.499658 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:49.499665 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:49.499823 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 19:06:49.499858 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 19:06:49.499920 28952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:49.496392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.158.67:42383 (local address 127.8.158.66:35165)
0929 19:06:49.496449 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:49.496451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:49.496506 (+ 55us) server_negotiation.cc:407] Connection header received
0929 19:06:49.496575 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:49.496576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:49.496585 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:49.496644 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:06:49.496814 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:49.497564 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:49.499037 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:49.499577 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:49.499658 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:49.499665 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:49.499823 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 19:06:49.499858 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 19:06:49.499966 28941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:49.496334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.66:35165 (local address 127.8.158.67:42383)
0929 19:06:49.496427 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:49.496439 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:49.496500 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:49.496658 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:49.496659 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:49.496661 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:06:49.496727 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:49.496729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:49.497663 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:49.497664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:49.498950 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:49.498953 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:49.499665 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:49.499666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:49.499741 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:49.499747 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 19:06:49.499803 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 19:06:49.499842 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":60}
I20260929 19:06:49.499984 28962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:49.496381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:42727 (local address 127.8.158.67:34227)
0929 19:06:49.496507 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:49.496517 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:06:49.496598 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:06:49.496774 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:06:49.496775 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:06:49.496776 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:06:49.496838 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:49.496839 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:49.497663 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:49.497664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:49.498950 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:06:49.498953 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:06:49.499666 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:06:49.499666 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:06:49.499741 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:49.499747 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 19:06:49.499803 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 19:06:49.499841 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":30,"thread_start_us":82,"threads_started":1}
I20260929 19:06:49.500326 28762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48f92f7058443bca54a4c94f51a4800" candidate_uuid: "c76ece8b4c72461281dedc35e7cbd0c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bd37eb0b27040b79130291494df1e50" is_pre_election: true
I20260929 19:06:49.500349 28631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48f92f7058443bca54a4c94f51a4800" candidate_uuid: "c76ece8b4c72461281dedc35e7cbd0c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c387b46178d24994ba4bba382cc3d7a2" is_pre_election: true
I20260929 19:06:49.500437 28762 raft_consensus.cc:2454] T a48f92f7058443bca54a4c94f51a4800 P 1bd37eb0b27040b79130291494df1e50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c76ece8b4c72461281dedc35e7cbd0c3 in term 0.
I20260929 19:06:49.500478 28631 raft_consensus.cc:2454] T a48f92f7058443bca54a4c94f51a4800 P c387b46178d24994ba4bba382cc3d7a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c76ece8b4c72461281dedc35e7cbd0c3 in term 0.
I20260929 19:06:49.500686 28828 leader_election.cc:302] T a48f92f7058443bca54a4c94f51a4800 P c76ece8b4c72461281dedc35e7cbd0c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bd37eb0b27040b79130291494df1e50, c76ece8b4c72461281dedc35e7cbd0c3; no voters:
I20260929 19:06:49.500805 28961 raft_consensus.cc:2781] T a48f92f7058443bca54a4c94f51a4800 P c76ece8b4c72461281dedc35e7cbd0c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:06:49.500857 28961 raft_consensus.cc:482] T a48f92f7058443bca54a4c94f51a4800 P c76ece8b4c72461281dedc35e7cbd0c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:06:49.500880 28961 raft_consensus.cc:3037] T a48f92f7058443bca54a4c94f51a4800 P c76ece8b4c72461281dedc35e7cbd0c3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:06:49.501464 28961 raft_consensus.cc:504] T a48f92f7058443bca54a4c94f51a4800 P c76ece8b4c72461281dedc35e7cbd0c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd37eb0b27040b79130291494df1e50" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 35165 } } peers { permanent_uuid: "c387b46178d24994ba4bba382cc3d7a2" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 42727 } } peers { permanent_uuid: "c76ece8b4c72461281dedc35e7cbd0c3" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 41719 } }
I20260929 19:06:49.501605 28961 leader_election.cc:288] T a48f92f7058443bca54a4c94f51a4800 P c76ece8b4c72461281dedc35e7cbd0c3 [CANDIDATE]: Term 1 election: Requested vote from peers 1bd37eb0b27040b79130291494df1e50 (127.8.158.66:35165), c387b46178d24994ba4bba382cc3d7a2 (127.8.158.65:42727)
I20260929 19:06:49.501717 28762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48f92f7058443bca54a4c94f51a4800" candidate_uuid: "c76ece8b4c72461281dedc35e7cbd0c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bd37eb0b27040b79130291494df1e50"
I20260929 19:06:49.501758 28631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48f92f7058443bca54a4c94f51a4800" candidate_uuid: "c76ece8b4c72461281dedc35e7cbd0c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c387b46178d24994ba4bba382cc3d7a2"
I20260929 19:06:49.501786 28762 raft_consensus.cc:3037] T a48f92f7058443bca54a4c94f51a4800 P 1bd37eb0b27040b79130291494df1e50 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:06:49.501859 28631 raft_consensus.cc:3037] T a48f92f7058443bca54a4c94f51a4800 P c387b46178d24994ba4bba382cc3d7a2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:06:49.502556 28762 raft_consensus.cc:2454] T a48f92f7058443bca54a4c94f51a4800 P 1bd37eb0b27040b79130291494df1e50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c76ece8b4c72461281dedc35e7cbd0c3 in term 1.
I20260929 19:06:49.502650 28631 raft_consensus.cc:2454] T a48f92f7058443bca54a4c94f51a4800 P c387b46178d24994ba4bba382cc3d7a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c76ece8b4c72461281dedc35e7cbd0c3 in term 1.
I20260929 19:06:49.502712 28828 leader_election.cc:302] T a48f92f7058443bca54a4c94f51a4800 P c76ece8b4c72461281dedc35e7cbd0c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bd37eb0b27040b79130291494df1e50, c76ece8b4c72461281dedc35e7cbd0c3; no voters:
I20260929 19:06:49.502842 28961 raft_consensus.cc:2781] T a48f92f7058443bca54a4c94f51a4800 P c76ece8b4c72461281dedc35e7cbd0c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:06:49.502957 28961 raft_consensus.cc:686] T a48f92f7058443bca54a4c94f51a4800 P c76ece8b4c72461281dedc35e7cbd0c3 [term 1 LEADER]: Becoming Leader. State: Replica: c76ece8b4c72461281dedc35e7cbd0c3, State: Running, Role: LEADER
I20260929 19:06:49.503046 28961 consensus_queue.cc:237] T a48f92f7058443bca54a4c94f51a4800 P c76ece8b4c72461281dedc35e7cbd0c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd37eb0b27040b79130291494df1e50" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 35165 } } peers { permanent_uuid: "c387b46178d24994ba4bba382cc3d7a2" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 42727 } } peers { permanent_uuid: "c76ece8b4c72461281dedc35e7cbd0c3" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 41719 } }
I20260929 19:06:49.503736 28489 catalog_manager.cc:5495] T a48f92f7058443bca54a4c94f51a4800 P c76ece8b4c72461281dedc35e7cbd0c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to c76ece8b4c72461281dedc35e7cbd0c3 (127.8.158.67). New cstate: current_term: 1 leader_uuid: "c76ece8b4c72461281dedc35e7cbd0c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bd37eb0b27040b79130291494df1e50" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 35165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c387b46178d24994ba4bba382cc3d7a2" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 42727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c76ece8b4c72461281dedc35e7cbd0c3" member_type: VOTER last_known_addr { host: "127.8.158.67" port: 41719 } health_report { overall_health: HEALTHY } } }
I20260929 19:06:49.508530 28951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:49.506233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60212 (local address 127.8.158.67:41719)
0929 19:06:49.506269 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:49.506271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:49.506279 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:06:49.506367 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:49.506367 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:49.506382 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:49.506453 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:06:49.506562 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:49.507009 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:49.507762 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:49.507969 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:49.508053 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:49.508226 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:06:49.508319 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:06:49.508410 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:49.508430 (+ 20us) server_negotiation.cc:299] Negotiation successful
0929 19:06:49.508448 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 19:06:49.509933 28631 raft_consensus.cc:1261] T a48f92f7058443bca54a4c94f51a4800 P c387b46178d24994ba4bba382cc3d7a2 [term 1 FOLLOWER]: Refusing update from remote peer c76ece8b4c72461281dedc35e7cbd0c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:06:49.509953 28762 raft_consensus.cc:1261] T a48f92f7058443bca54a4c94f51a4800 P 1bd37eb0b27040b79130291494df1e50 [term 1 FOLLOWER]: Refusing update from remote peer c76ece8b4c72461281dedc35e7cbd0c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:06:49.510294 28963 consensus_queue.cc:1035] T a48f92f7058443bca54a4c94f51a4800 P c76ece8b4c72461281dedc35e7cbd0c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c387b46178d24994ba4bba382cc3d7a2" member_type: VOTER last_known_addr { host: "127.8.158.65" port: 42727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:06:49.510422 28961 consensus_queue.cc:1035] T a48f92f7058443bca54a4c94f51a4800 P c76ece8b4c72461281dedc35e7cbd0c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bd37eb0b27040b79130291494df1e50" member_type: VOTER last_known_addr { host: "127.8.158.66" port: 35165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:06:49.512734 28966 mvcc.cc:205] Tried to move back new op lower bound from 7334743283750862848 to 7334743283725176832. Current Snapshot: MvccSnapshot[applied={T|T < 7334743283750862848}]
I20260929 19:06:49.513113 28967 mvcc.cc:205] Tried to move back new op lower bound from 7334743283750862848 to 7334743283725176832. Current Snapshot: MvccSnapshot[applied={T|T < 7334743283750862848}]
I20260929 19:06:49.513440 28968 mvcc.cc:205] Tried to move back new op lower bound from 7334743283750862848 to 7334743283725176832. Current Snapshot: MvccSnapshot[applied={T|T < 7334743283750862848}]
I20260929 19:06:49.515218 28489 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:06:49.515317 28489 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:06:49.515987 28489 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cad2eda191d34576922224bf3cc3b6d2: Sending DeleteTablet for 3 replicas of tablet a48f92f7058443bca54a4c94f51a4800
I20260929 19:06:49.516332 28611 tablet_service.cc:1514] Processing DeleteTablet for tablet a48f92f7058443bca54a4c94f51a4800 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:06:49 UTC) from {username='slave'} at 127.0.0.1:42998
I20260929 19:06:49.516330 28742 tablet_service.cc:1514] Processing DeleteTablet for tablet a48f92f7058443bca54a4c94f51a4800 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:06:49 UTC) from {username='slave'} at 127.0.0.1:51114
I20260929 19:06:49.516346 28873 tablet_service.cc:1514] Processing DeleteTablet for tablet a48f92f7058443bca54a4c94f51a4800 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:06:49 UTC) from {username='slave'} at 127.0.0.1:60196
I20260929 19:06:49.516604 28976 tablet_replica.cc:330] T a48f92f7058443bca54a4c94f51a4800 P c387b46178d24994ba4bba382cc3d7a2: stopping tablet replica
I20260929 19:06:49.516702 28976 raft_consensus.cc:2229] T a48f92f7058443bca54a4c94f51a4800 P c387b46178d24994ba4bba382cc3d7a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:06:49.516798 28976 raft_consensus.cc:2258] T a48f92f7058443bca54a4c94f51a4800 P c387b46178d24994ba4bba382cc3d7a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:06:49.516847 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 28549
I20260929 19:06:49.516942 28977 tablet_replica.cc:330] T a48f92f7058443bca54a4c94f51a4800 P 1bd37eb0b27040b79130291494df1e50: stopping tablet replica
I20260929 19:06:49.517023 28977 raft_consensus.cc:2229] T a48f92f7058443bca54a4c94f51a4800 P 1bd37eb0b27040b79130291494df1e50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:06:49.517108 28977 raft_consensus.cc:2258] T a48f92f7058443bca54a4c94f51a4800 P 1bd37eb0b27040b79130291494df1e50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:06:49.517532 28977 ts_tablet_manager.cc:1905] T a48f92f7058443bca54a4c94f51a4800 P 1bd37eb0b27040b79130291494df1e50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:06:49.518807 28978 tablet_replica.cc:330] T a48f92f7058443bca54a4c94f51a4800 P c76ece8b4c72461281dedc35e7cbd0c3: stopping tablet replica
I20260929 19:06:49.518908 28978 raft_consensus.cc:2229] T a48f92f7058443bca54a4c94f51a4800 P c76ece8b4c72461281dedc35e7cbd0c3 [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:06:49.519021 28978 raft_consensus.cc:2258] T a48f92f7058443bca54a4c94f51a4800 P c76ece8b4c72461281dedc35e7cbd0c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:06:49.518909 28977 ts_tablet_manager.cc:1918] T a48f92f7058443bca54a4c94f51a4800 P 1bd37eb0b27040b79130291494df1e50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:06:49.519157 28977 log.cc:1196] T a48f92f7058443bca54a4c94f51a4800 P 1bd37eb0b27040b79130291494df1e50: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-1/wal/wals/a48f92f7058443bca54a4c94f51a4800
I20260929 19:06:49.519430 28978 ts_tablet_manager.cc:1905] T a48f92f7058443bca54a4c94f51a4800 P c76ece8b4c72461281dedc35e7cbd0c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:06:49.519542 28977 ts_tablet_manager.cc:1939] T a48f92f7058443bca54a4c94f51a4800 P 1bd37eb0b27040b79130291494df1e50: Deleting consensus metadata
I20260929 19:06:49.519984 28475 catalog_manager.cc:4841] TS 1bd37eb0b27040b79130291494df1e50 (127.8.158.66:35165): tablet a48f92f7058443bca54a4c94f51a4800 (table test-table [id=b840c3885ada42809bdc5f8ed5a6cfa3]) successfully deleted
I20260929 19:06:49.521045 28978 ts_tablet_manager.cc:1918] T a48f92f7058443bca54a4c94f51a4800 P c76ece8b4c72461281dedc35e7cbd0c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:06:49.521142 28978 log.cc:1196] T a48f92f7058443bca54a4c94f51a4800 P c76ece8b4c72461281dedc35e7cbd0c3: Deleting WAL directory at /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708573736807-8825-0/minicluster-data/ts-2/wal/wals/a48f92f7058443bca54a4c94f51a4800
I20260929 19:06:49.521371 28978 ts_tablet_manager.cc:1939] T a48f92f7058443bca54a4c94f51a4800 P c76ece8b4c72461281dedc35e7cbd0c3: Deleting consensus metadata
I20260929 19:06:49.524092 28473 catalog_manager.cc:4841] TS c76ece8b4c72461281dedc35e7cbd0c3 (127.8.158.67:41719): tablet a48f92f7058443bca54a4c94f51a4800 (table test-table [id=b840c3885ada42809bdc5f8ed5a6cfa3]) successfully deleted
W20260929 19:06:49.524159 28474 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.158.65:42727 (error 108)
I20260929 19:06:49.524739 28947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:49.524617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.65:42727 (local address 127.0.0.1:43006)
0929 19:06:49.524670 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0929 19:06:49.524714 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.65:42727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 19:06:49.524755 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 28680
W20260929 19:06:49.526974 28474 catalog_manager.cc:4584] TS c387b46178d24994ba4bba382cc3d7a2 (127.8.158.65:42727): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a48f92f7058443bca54a4c94f51a4800: Network error: Client connection negotiation failed: client connection to 127.8.158.65:42727: connect: Connection refused (error 111)
I20260929 19:06:49.531107 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 28811
I20260929 19:06:49.535596 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 28458
2026-09-29T19:06:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (631 ms)
[----------] 6 tests from EncryptionPolicyTest (8002 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708573736807-8825-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:06:49 dist-test-slave-nmsz krb5kdc[28982](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:06:49 dist-test-slave-nmsz krb5kdc[28982](info): set up 2 sockets
Sep 29 19:06:49 dist-test-slave-nmsz krb5kdc[28982](info): commencing operation
krb5kdc: starting...
W20260929 19:06:50.642254 8825 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Sep 29 19:06:50 dist-test-slave-nmsz krb5kdc[28982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:06:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:06:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708573736807-8825-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708573736807-8825-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 19:06:50.754594 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39303
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708573736807-8825-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:06:50.833900 28995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:06:50.834117 28995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:06:50.834154 28995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:06:50.835300 28995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:06:50.835369 28995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:06:50.835395 28995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:06:50.835415 28995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:06:50.835448 28995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:06:50.836867 28995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39303
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708573736807-8825-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:06:50.837132 28995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:06:50.837343 28995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:06:50.841799 29001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:50.841779 29003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:50.841843 29000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:50.842170 28995 server_base.cc:1034] running on GCE node
I20260929 19:06:50.842427 28995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:06:50.842686 28995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:06:50.843842 28995 hybrid_clock.cc:648] HybridClock initialized: now 1790708810843836 us; error 36 us; skew 500 ppm
Sep 29 19:06:50 dist-test-slave-nmsz krb5kdc[28982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:06:50.846233 28995 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 19:06:50.846702 28995 webserver.cc:460] Webserver started at http://127.0.0.1:36297/ using document root <none> and password file <none>
I20260929 19:06:50.846938 28995 fs_manager.cc:359] Metadata directory not provided
I20260929 19:06:50.847004 28995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:06:50.847134 28995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:06:50.848075 28995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "1397dce252bf49aa842cade8ee7deab1"
format_stamp: "Formatted at 2026-09-29 19:06:50 on dist-test-slave-nmsz"
server_key: "11b833f495fc0e95972be998210dacf7"
server_key_iv: "85e424c6a85846d924ceea4991445909"
server_key_version: "encryptionkey@0"
I20260929 19:06:50.848412 28995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "1397dce252bf49aa842cade8ee7deab1"
format_stamp: "Formatted at 2026-09-29 19:06:50 on dist-test-slave-nmsz"
server_key: "11b833f495fc0e95972be998210dacf7"
server_key_iv: "85e424c6a85846d924ceea4991445909"
server_key_version: "encryptionkey@0"
I20260929 19:06:50.849679 28995 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:06:50.850441 29010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:50.850618 28995 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 19:06:50.850711 28995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "1397dce252bf49aa842cade8ee7deab1"
format_stamp: "Formatted at 2026-09-29 19:06:50 on dist-test-slave-nmsz"
server_key: "11b833f495fc0e95972be998210dacf7"
server_key_iv: "85e424c6a85846d924ceea4991445909"
server_key_version: "encryptionkey@0"
I20260929 19:06:50.850795 28995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:06:50.863484 28995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:06:50.865145 28995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:06:50.865293 28995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:06:50.869210 28995 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39303
I20260929 19:06:50.869240 29062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39303 every 8 connection(s)
I20260929 19:06:50.869547 28995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:06:50.870473 29063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:06:50.872428 29063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1397dce252bf49aa842cade8ee7deab1: Bootstrap starting.
I20260929 19:06:50.872903 29063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1397dce252bf49aa842cade8ee7deab1: Neither blocks nor log segments found. Creating new log.
I20260929 19:06:50.873127 29063 log.cc:824] T 00000000000000000000000000000000 P 1397dce252bf49aa842cade8ee7deab1: Log is configured to *not* fsync() on all Append() calls
I20260929 19:06:50.873708 29063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1397dce252bf49aa842cade8ee7deab1: No bootstrap required, opened a new log
I20260929 19:06:50.875245 29063 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1397dce252bf49aa842cade8ee7deab1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1397dce252bf49aa842cade8ee7deab1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39303 } }
I20260929 19:06:50.875370 29063 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1397dce252bf49aa842cade8ee7deab1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:06:50.875391 29063 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1397dce252bf49aa842cade8ee7deab1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1397dce252bf49aa842cade8ee7deab1, State: Initialized, Role: FOLLOWER
I20260929 19:06:50.875483 29063 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1397dce252bf49aa842cade8ee7deab1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1397dce252bf49aa842cade8ee7deab1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39303 } }
I20260929 19:06:50.875531 29063 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1397dce252bf49aa842cade8ee7deab1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:06:50.875559 29063 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1397dce252bf49aa842cade8ee7deab1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:06:50.875591 29063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1397dce252bf49aa842cade8ee7deab1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:06:50.876022 29063 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1397dce252bf49aa842cade8ee7deab1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1397dce252bf49aa842cade8ee7deab1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39303 } }
I20260929 19:06:50.876099 29063 leader_election.cc:302] T 00000000000000000000000000000000 P 1397dce252bf49aa842cade8ee7deab1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1397dce252bf49aa842cade8ee7deab1; no voters:
I20260929 19:06:50.876209 29063 leader_election.cc:288] T 00000000000000000000000000000000 P 1397dce252bf49aa842cade8ee7deab1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:06:50.876260 29066 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1397dce252bf49aa842cade8ee7deab1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:06:50.876421 29066 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1397dce252bf49aa842cade8ee7deab1 [term 1 LEADER]: Becoming Leader. State: Replica: 1397dce252bf49aa842cade8ee7deab1, State: Running, Role: LEADER
I20260929 19:06:50.876462 29063 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1397dce252bf49aa842cade8ee7deab1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:06:50.876503 29066 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1397dce252bf49aa842cade8ee7deab1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1397dce252bf49aa842cade8ee7deab1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39303 } }
I20260929 19:06:50.876892 29068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1397dce252bf49aa842cade8ee7deab1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1397dce252bf49aa842cade8ee7deab1. Latest consensus state: current_term: 1 leader_uuid: "1397dce252bf49aa842cade8ee7deab1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1397dce252bf49aa842cade8ee7deab1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39303 } } }
I20260929 19:06:50.876978 29068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1397dce252bf49aa842cade8ee7deab1 [sys.catalog]: This master's current role is: LEADER
I20260929 19:06:50.877116 29067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1397dce252bf49aa842cade8ee7deab1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1397dce252bf49aa842cade8ee7deab1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1397dce252bf49aa842cade8ee7deab1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39303 } } }
I20260929 19:06:50.877256 29067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1397dce252bf49aa842cade8ee7deab1 [sys.catalog]: This master's current role is: LEADER
I20260929 19:06:50.877553 29074 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:06:50.878039 29074 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:06:50.878753 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 28995
I20260929 19:06:50.878885 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:06:50.879070 8825 external_mini_cluster.cc:1428] Setting key 3b9219debfd624bfbd01c3b20b2786dd
I20260929 19:06:50.879549 29074 catalog_manager.cc:1347] Generated new cluster ID: e51ba2cd2b5c439e9ad2616314e47ba5
I20260929 19:06:50.879597 29074 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 19:06:50 dist-test-slave-nmsz krb5kdc[28982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 19:06:50.890269 29086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:50.879667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49467 (local address 127.0.0.1:39303)
0929 19:06:50.879817 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:50.879825 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:50.879852 (+ 27us) server_negotiation.cc:407] Connection header received
0929 19:06:50.879948 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:50.879954 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:50.880002 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:50.880122 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:50.880871 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:50.882248 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:50.883333 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:50.883895 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:50.883996 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:50.886706 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:50.886710 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:50.886710 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:06:50.886721 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:50.888662 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:50.889076 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:50.889079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:50.889079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:50.889180 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:50.889412 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:50.889413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:50.889413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:50.889696 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:50.889852 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:50.889961 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 19:06:50.890055 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":68,"threads_started":1}
I20260929 19:06:50.890703 29074 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:06:50.891350 29074 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:06:50.891396 8825 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:06:50.899858 29086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:50.892759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56265 (local address 127.0.0.1:39303)
0929 19:06:50.892793 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:50.892795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:50.892888 (+ 93us) server_negotiation.cc:407] Connection header received
0929 19:06:50.892930 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:50.892930 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:50.892939 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:50.893004 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:50.893542 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:50.894303 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:50.895289 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:50.895614 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:50.895682 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:50.896938 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:50.896940 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:50.896940 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:06:50.896947 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:50.898465 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:50.898856 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:50.898857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:50.898857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:50.898912 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:50.899142 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:50.899143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:50.899143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:50.899273 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:50.899383 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:50.899741 (+ 358us) server_negotiation.cc:299] Negotiation successful
0929 19:06:50.899797 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":9,"server-negotiator.queue_time_us":16}
I20260929 19:06:50.899942 29074 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1397dce252bf49aa842cade8ee7deab1: Generated new TSK 0
I20260929 19:06:50.900081 29074 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:06:50.904246 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 28995
W20260929 19:06:50.908511 28980 connection.cc:535] client connection to 127.0.0.1:39303 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39303): Connection reset by peer (error 104)
2026-09-29T19:06:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1367 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708573736807-8825-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:06:50 dist-test-slave-nmsz krb5kdc[29094](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:06:50 dist-test-slave-nmsz krb5kdc[29094](info): set up 2 sockets
Sep 29 19:06:50 dist-test-slave-nmsz krb5kdc[29094](info): commencing operation
krb5kdc: starting...
W20260929 19:06:51.930171 8825 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Sep 29 19:06:52 dist-test-slave-nmsz krb5kdc[29094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:06:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:06:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708573736807-8825-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708573736807-8825-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 19:06:52.045234 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42537
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708573736807-8825-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:06:52.124395 29107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:06:52.124583 29107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:06:52.124604 29107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:06:52.125680 29107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:06:52.125717 29107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:06:52.125731 29107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:06:52.125744 29107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:06:52.125756 29107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:06:52.127157 29107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42537
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708573736807-8825-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:06:52.127338 29107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:06:52.127489 29107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:06:52.132170 29113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:52.132145 29112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:52.132185 29115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:52.132495 29107 server_base.cc:1034] running on GCE node
I20260929 19:06:52.132727 29107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:06:52.132997 29107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:06:52.134159 29107 hybrid_clock.cc:648] HybridClock initialized: now 1790708812134146 us; error 43 us; skew 500 ppm
Sep 29 19:06:52 dist-test-slave-nmsz krb5kdc[29094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:06:52.140609 29107 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 19:06:52.140980 29107 webserver.cc:460] Webserver started at http://127.0.0.1:34109/ using document root <none> and password file <none>
I20260929 19:06:52.141162 29107 fs_manager.cc:359] Metadata directory not provided
I20260929 19:06:52.141201 29107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:06:52.141284 29107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:06:52.142216 29107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "b84ec29a7d32485ebe057c1b0517686e"
format_stamp: "Formatted at 2026-09-29 19:06:52 on dist-test-slave-nmsz"
server_key: "258588b24bc025b7295715d3731a3697"
server_key_iv: "75b9e332d01134bb8df3e1fd09651426"
server_key_version: "encryptionkey@0"
I20260929 19:06:52.142578 29107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "b84ec29a7d32485ebe057c1b0517686e"
format_stamp: "Formatted at 2026-09-29 19:06:52 on dist-test-slave-nmsz"
server_key: "258588b24bc025b7295715d3731a3697"
server_key_iv: "75b9e332d01134bb8df3e1fd09651426"
server_key_version: "encryptionkey@0"
I20260929 19:06:52.143776 29107 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:06:52.144444 29122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:52.144618 29107 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:06:52.144696 29107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "b84ec29a7d32485ebe057c1b0517686e"
format_stamp: "Formatted at 2026-09-29 19:06:52 on dist-test-slave-nmsz"
server_key: "258588b24bc025b7295715d3731a3697"
server_key_iv: "75b9e332d01134bb8df3e1fd09651426"
server_key_version: "encryptionkey@0"
I20260929 19:06:52.144747 29107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:06:52.156342 29107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:06:52.158104 29107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:06:52.158205 29107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:06:52.161877 29107 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42537
I20260929 19:06:52.161954 29174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42537 every 8 connection(s)
I20260929 19:06:52.162241 29107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:06:52.163231 29175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:06:52.165300 29175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b84ec29a7d32485ebe057c1b0517686e: Bootstrap starting.
I20260929 19:06:52.165851 29175 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b84ec29a7d32485ebe057c1b0517686e: Neither blocks nor log segments found. Creating new log.
I20260929 19:06:52.166141 29175 log.cc:824] T 00000000000000000000000000000000 P b84ec29a7d32485ebe057c1b0517686e: Log is configured to *not* fsync() on all Append() calls
I20260929 19:06:52.166827 29175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b84ec29a7d32485ebe057c1b0517686e: No bootstrap required, opened a new log
I20260929 19:06:52.168316 29175 raft_consensus.cc:348] T 00000000000000000000000000000000 P b84ec29a7d32485ebe057c1b0517686e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b84ec29a7d32485ebe057c1b0517686e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42537 } }
I20260929 19:06:52.168469 29175 raft_consensus.cc:374] T 00000000000000000000000000000000 P b84ec29a7d32485ebe057c1b0517686e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:06:52.168516 29175 raft_consensus.cc:729] T 00000000000000000000000000000000 P b84ec29a7d32485ebe057c1b0517686e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b84ec29a7d32485ebe057c1b0517686e, State: Initialized, Role: FOLLOWER
I20260929 19:06:52.168681 29175 consensus_queue.cc:260] T 00000000000000000000000000000000 P b84ec29a7d32485ebe057c1b0517686e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b84ec29a7d32485ebe057c1b0517686e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42537 } }
I20260929 19:06:52.168780 29175 raft_consensus.cc:388] T 00000000000000000000000000000000 P b84ec29a7d32485ebe057c1b0517686e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:06:52.168823 29175 raft_consensus.cc:482] T 00000000000000000000000000000000 P b84ec29a7d32485ebe057c1b0517686e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:06:52.168879 29175 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b84ec29a7d32485ebe057c1b0517686e [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:06:52.169358 29175 raft_consensus.cc:504] T 00000000000000000000000000000000 P b84ec29a7d32485ebe057c1b0517686e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b84ec29a7d32485ebe057c1b0517686e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42537 } }
I20260929 19:06:52.169473 29175 leader_election.cc:302] T 00000000000000000000000000000000 P b84ec29a7d32485ebe057c1b0517686e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b84ec29a7d32485ebe057c1b0517686e; no voters:
I20260929 19:06:52.169665 29175 leader_election.cc:288] T 00000000000000000000000000000000 P b84ec29a7d32485ebe057c1b0517686e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:06:52.169687 29178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b84ec29a7d32485ebe057c1b0517686e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:06:52.169677 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 29107
I20260929 19:06:52.169797 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:06:52.169798 29178 raft_consensus.cc:686] T 00000000000000000000000000000000 P b84ec29a7d32485ebe057c1b0517686e [term 1 LEADER]: Becoming Leader. State: Replica: b84ec29a7d32485ebe057c1b0517686e, State: Running, Role: LEADER
I20260929 19:06:52.169870 29178 consensus_queue.cc:237] T 00000000000000000000000000000000 P b84ec29a7d32485ebe057c1b0517686e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b84ec29a7d32485ebe057c1b0517686e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42537 } }
I20260929 19:06:52.169922 29175 sys_catalog.cc:564] T 00000000000000000000000000000000 P b84ec29a7d32485ebe057c1b0517686e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:06:52.170009 8825 external_mini_cluster.cc:1428] Setting key 0fafa29861ea0f9d037d3ff959301cbd
I20260929 19:06:52.170411 29179 sys_catalog.cc:455] T 00000000000000000000000000000000 P b84ec29a7d32485ebe057c1b0517686e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b84ec29a7d32485ebe057c1b0517686e. Latest consensus state: current_term: 1 leader_uuid: "b84ec29a7d32485ebe057c1b0517686e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b84ec29a7d32485ebe057c1b0517686e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42537 } } }
I20260929 19:06:52.170930 29179 sys_catalog.cc:458] T 00000000000000000000000000000000 P b84ec29a7d32485ebe057c1b0517686e [sys.catalog]: This master's current role is: LEADER
I20260929 19:06:52.170169 29178 sys_catalog.cc:455] T 00000000000000000000000000000000 P b84ec29a7d32485ebe057c1b0517686e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b84ec29a7d32485ebe057c1b0517686e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b84ec29a7d32485ebe057c1b0517686e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42537 } } }
I20260929 19:06:52.171139 29178 sys_catalog.cc:458] T 00000000000000000000000000000000 P b84ec29a7d32485ebe057c1b0517686e [sys.catalog]: This master's current role is: LEADER
I20260929 19:06:52.171273 29193 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:06:52.171904 29193 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:06:52.173483 29193 catalog_manager.cc:1347] Generated new cluster ID: 12694098792d489c929ce46417206cf6
I20260929 19:06:52.173537 29193 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:06:52.175674 29187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:52.170504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42483 (local address 127.0.0.1:42537)
0929 19:06:52.171114 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:52.171123 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:52.171156 (+ 33us) server_negotiation.cc:407] Connection header received
0929 19:06:52.171270 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:52.171277 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:52.171333 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:52.171470 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:52.171707 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:52.172800 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:52.173820 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:52.174252 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:52.174397 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:52.175091 (+ 694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:52.175095 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:52.175095 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:06:52.175267 (+ 172us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:52.175296 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:06:52.175306 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:52.175399 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:52.175477 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 19:06:52.175507 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":77,"threads_started":1}
I20260929 19:06:52.179286 8825 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:06:52.185211 29187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:52.180642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35127 (local address 127.0.0.1:42537)
0929 19:06:52.180682 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:52.180684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:52.182503 (+ 1819us) server_negotiation.cc:407] Connection header received
0929 19:06:52.182576 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:52.182577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:52.182587 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:52.182641 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:52.182791 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:52.183267 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:52.183993 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:52.184218 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:52.184305 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:52.184967 (+ 662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:52.184969 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:52.184969 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:06:52.184974 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:52.184986 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:06:52.184990 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:52.185043 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:52.185109 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 19:06:52.185120 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 19:06:52.188102 29193 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:06:52.188782 29193 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:06:52.205463 29193 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b84ec29a7d32485ebe057c1b0517686e: Generated new TSK 0
I20260929 19:06:52.205632 29193 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:06:52.212852 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 29107
W20260929 19:06:52.217026 29092 connection.cc:535] client connection to 127.0.0.1:42537 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42537): Connection reset by peer (error 104)
2026-09-29T19:06:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1308 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708573736807-8825-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:06:52 dist-test-slave-nmsz krb5kdc[29206](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:06:52 dist-test-slave-nmsz krb5kdc[29206](info): set up 2 sockets
Sep 29 19:06:52 dist-test-slave-nmsz krb5kdc[29206](info): commencing operation
krb5kdc: starting...
W20260929 19:06:53.246225 8825 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Sep 29 19:06:53 dist-test-slave-nmsz krb5kdc[29206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:06:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:06:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708573736807-8825-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708573736807-8825-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 19:06:53.361585 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42449
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708573736807-8825-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:06:53.440392 29219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:06:53.440611 29219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:06:53.440649 29219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:06:53.441759 29219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:06:53.441823 29219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:06:53.441848 29219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:06:53.441877 29219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:06:53.441906 29219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:06:53.443413 29219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42449
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708573736807-8825-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:06:53.443676 29219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:06:53.443889 29219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:06:53.448766 29224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:53.448778 29227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:53.448784 29225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:53.448794 29219 server_base.cc:1034] running on GCE node
I20260929 19:06:53.449204 29219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:06:53.449487 29219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:06:53.450642 29219 hybrid_clock.cc:648] HybridClock initialized: now 1790708813450631 us; error 45 us; skew 500 ppm
Sep 29 19:06:53 dist-test-slave-nmsz krb5kdc[29206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:06:53.452951 29219 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 19:06:53.453385 29219 webserver.cc:460] Webserver started at http://127.0.0.1:40521/ using document root <none> and password file <none>
I20260929 19:06:53.453606 29219 fs_manager.cc:359] Metadata directory not provided
I20260929 19:06:53.453671 29219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:06:53.453816 29219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:06:53.454842 29219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "77107f771a2e45da85070cc7d1841028"
format_stamp: "Formatted at 2026-09-29 19:06:53 on dist-test-slave-nmsz"
server_key: "095603916f86037c6e81686f6b824e82"
server_key_iv: "9473d45eff001d7fcffaddfffd6b7b1d"
server_key_version: "encryptionkey@0"
I20260929 19:06:53.455186 29219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "77107f771a2e45da85070cc7d1841028"
format_stamp: "Formatted at 2026-09-29 19:06:53 on dist-test-slave-nmsz"
server_key: "095603916f86037c6e81686f6b824e82"
server_key_iv: "9473d45eff001d7fcffaddfffd6b7b1d"
server_key_version: "encryptionkey@0"
I20260929 19:06:53.456416 29219 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:06:53.457113 29234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:53.457309 29219 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:06:53.457419 29219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "77107f771a2e45da85070cc7d1841028"
format_stamp: "Formatted at 2026-09-29 19:06:53 on dist-test-slave-nmsz"
server_key: "095603916f86037c6e81686f6b824e82"
server_key_iv: "9473d45eff001d7fcffaddfffd6b7b1d"
server_key_version: "encryptionkey@0"
I20260929 19:06:53.457496 29219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:06:53.460701 29219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:06:53.462544 29219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:06:53.462684 29219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:06:53.466609 29219 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42449
I20260929 19:06:53.466655 29286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42449 every 8 connection(s)
I20260929 19:06:53.466976 29219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:06:53.467839 29287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:06:53.469792 29287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77107f771a2e45da85070cc7d1841028: Bootstrap starting.
I20260929 19:06:53.470257 29287 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77107f771a2e45da85070cc7d1841028: Neither blocks nor log segments found. Creating new log.
I20260929 19:06:53.470564 29287 log.cc:824] T 00000000000000000000000000000000 P 77107f771a2e45da85070cc7d1841028: Log is configured to *not* fsync() on all Append() calls
I20260929 19:06:53.471136 29287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77107f771a2e45da85070cc7d1841028: No bootstrap required, opened a new log
I20260929 19:06:53.472622 29287 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77107f771a2e45da85070cc7d1841028 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77107f771a2e45da85070cc7d1841028" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42449 } }
I20260929 19:06:53.472744 29287 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77107f771a2e45da85070cc7d1841028 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:06:53.472765 29287 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77107f771a2e45da85070cc7d1841028 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77107f771a2e45da85070cc7d1841028, State: Initialized, Role: FOLLOWER
I20260929 19:06:53.472890 29287 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77107f771a2e45da85070cc7d1841028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77107f771a2e45da85070cc7d1841028" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42449 } }
I20260929 19:06:53.472973 29287 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77107f771a2e45da85070cc7d1841028 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:06:53.472996 29287 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77107f771a2e45da85070cc7d1841028 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:06:53.473029 29287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77107f771a2e45da85070cc7d1841028 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:06:53.473479 29287 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77107f771a2e45da85070cc7d1841028 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77107f771a2e45da85070cc7d1841028" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42449 } }
I20260929 19:06:53.473563 29287 leader_election.cc:302] T 00000000000000000000000000000000 P 77107f771a2e45da85070cc7d1841028 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77107f771a2e45da85070cc7d1841028; no voters:
I20260929 19:06:53.473701 29287 leader_election.cc:288] T 00000000000000000000000000000000 P 77107f771a2e45da85070cc7d1841028 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:06:53.473728 29290 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77107f771a2e45da85070cc7d1841028 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:06:53.473845 29290 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77107f771a2e45da85070cc7d1841028 [term 1 LEADER]: Becoming Leader. State: Replica: 77107f771a2e45da85070cc7d1841028, State: Running, Role: LEADER
I20260929 19:06:53.473948 29287 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77107f771a2e45da85070cc7d1841028 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:06:53.473979 29290 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77107f771a2e45da85070cc7d1841028 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77107f771a2e45da85070cc7d1841028" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42449 } }
I20260929 19:06:53.474383 29292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77107f771a2e45da85070cc7d1841028 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77107f771a2e45da85070cc7d1841028. Latest consensus state: current_term: 1 leader_uuid: "77107f771a2e45da85070cc7d1841028" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77107f771a2e45da85070cc7d1841028" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42449 } } }
I20260929 19:06:53.474496 29291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77107f771a2e45da85070cc7d1841028 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77107f771a2e45da85070cc7d1841028" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77107f771a2e45da85070cc7d1841028" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42449 } } }
I20260929 19:06:53.474538 29292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77107f771a2e45da85070cc7d1841028 [sys.catalog]: This master's current role is: LEADER
I20260929 19:06:53.474874 29291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77107f771a2e45da85070cc7d1841028 [sys.catalog]: This master's current role is: LEADER
I20260929 19:06:53.474977 29299 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:06:53.475490 29299 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:06:53.475621 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 29219
I20260929 19:06:53.475728 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:06:53.475921 8825 external_mini_cluster.cc:1428] Setting key 237c29bb45ac295644ab424541a864a8
I20260929 19:06:53.476955 29299 catalog_manager.cc:1347] Generated new cluster ID: b66db37342b344d5a721022b6e104d6f
I20260929 19:06:53.477011 29299 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:06:53.478081 29309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:53.476312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39829 (local address 127.0.0.1:42449)
0929 19:06:53.476477 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:53.476486 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:53.476584 (+ 98us) server_negotiation.cc:407] Connection header received
0929 19:06:53.476726 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:53.476733 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:53.476764 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:53.476897 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:53.477683 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:53.477686 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:53.477687 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:06:53.477691 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:53.477729 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:06:53.477743 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:53.477813 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:53.477878 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 19:06:53.477917 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":84,"threads_started":1}
I20260929 19:06:53.478979 8825 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:06:53.483359 29309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:53.480361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33391 (local address 127.0.0.1:42449)
0929 19:06:53.480391 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:53.480393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:53.482481 (+ 2088us) server_negotiation.cc:407] Connection header received
0929 19:06:53.482511 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:53.482512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:53.482517 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:53.482585 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:53.483141 (+ 556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:53.483142 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:53.483142 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:06:53.483146 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:53.483160 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:06:53.483164 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:53.483226 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:53.483278 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 19:06:53.483290 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 19:06:53.485859 29299 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:06:53.486004 29299 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:06:53.489607 29299 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77107f771a2e45da85070cc7d1841028: Generated new TSK 0
I20260929 19:06:53.489761 29299 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:06:53.495909 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 29219
2026-09-29T19:06:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1282 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708573736807-8825-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:06:53 dist-test-slave-nmsz krb5kdc[29318](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:06:53 dist-test-slave-nmsz krb5kdc[29318](info): set up 2 sockets
Sep 29 19:06:53 dist-test-slave-nmsz krb5kdc[29318](info): commencing operation
krb5kdc: starting...
W20260929 19:06:54.522260 8825 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Sep 29 19:06:54 dist-test-slave-nmsz krb5kdc[29318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:06:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:06:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708573736807-8825-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708573736807-8825-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 19:06:54.633832 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44115
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708573736807-8825-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:06:54.710330 29331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:06:54.710543 29331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:06:54.710580 29331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:06:54.711697 29331 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:06:54.711761 29331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:06:54.711787 29331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:06:54.711817 29331 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:06:54.711840 29331 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:06:54.713310 29331 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708573736807-8825-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:06:54.713559 29331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:06:54.713784 29331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:06:54.718500 29339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:54.718515 29336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:54.718515 29337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:54.718596 29331 server_base.cc:1034] running on GCE node
I20260929 19:06:54.718916 29331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:06:54.719199 29331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:06:54.720381 29331 hybrid_clock.cc:648] HybridClock initialized: now 1790708814720367 us; error 24 us; skew 500 ppm
Sep 29 19:06:54 dist-test-slave-nmsz krb5kdc[29318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:06:54.722617 29331 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 19:06:54.722998 29331 webserver.cc:460] Webserver started at http://127.0.0.1:45415/ using document root <none> and password file <none>
I20260929 19:06:54.723222 29331 fs_manager.cc:359] Metadata directory not provided
I20260929 19:06:54.723289 29331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:06:54.723433 29331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:06:54.724445 29331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "8af2c060feb946169db3fa38b38468ce"
format_stamp: "Formatted at 2026-09-29 19:06:54 on dist-test-slave-nmsz"
server_key: "7e7d7d56098ff33c07ac70e30004ec40"
server_key_iv: "64bf7b396efdd25d882c86c1ed06d110"
server_key_version: "encryptionkey@0"
I20260929 19:06:54.724789 29331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "8af2c060feb946169db3fa38b38468ce"
format_stamp: "Formatted at 2026-09-29 19:06:54 on dist-test-slave-nmsz"
server_key: "7e7d7d56098ff33c07ac70e30004ec40"
server_key_iv: "64bf7b396efdd25d882c86c1ed06d110"
server_key_version: "encryptionkey@0"
I20260929 19:06:54.726070 29331 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:06:54.726800 29346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:54.726979 29331 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:54.727089 29331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "8af2c060feb946169db3fa38b38468ce"
format_stamp: "Formatted at 2026-09-29 19:06:54 on dist-test-slave-nmsz"
server_key: "7e7d7d56098ff33c07ac70e30004ec40"
server_key_iv: "64bf7b396efdd25d882c86c1ed06d110"
server_key_version: "encryptionkey@0"
I20260929 19:06:54.727174 29331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:06:54.745030 29331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:06:54.746861 29331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:06:54.747027 29331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:06:54.750748 29331 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44115
I20260929 19:06:54.750780 29398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44115 every 8 connection(s)
I20260929 19:06:54.751094 29331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:06:54.752027 29399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:06:54.754052 29399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8af2c060feb946169db3fa38b38468ce: Bootstrap starting.
I20260929 19:06:54.754637 29399 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8af2c060feb946169db3fa38b38468ce: Neither blocks nor log segments found. Creating new log.
I20260929 19:06:54.754856 29399 log.cc:824] T 00000000000000000000000000000000 P 8af2c060feb946169db3fa38b38468ce: Log is configured to *not* fsync() on all Append() calls
I20260929 19:06:54.755487 29399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8af2c060feb946169db3fa38b38468ce: No bootstrap required, opened a new log
I20260929 19:06:54.756973 29399 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8af2c060feb946169db3fa38b38468ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8af2c060feb946169db3fa38b38468ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44115 } }
I20260929 19:06:54.757098 29399 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8af2c060feb946169db3fa38b38468ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:06:54.757119 29399 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8af2c060feb946169db3fa38b38468ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8af2c060feb946169db3fa38b38468ce, State: Initialized, Role: FOLLOWER
I20260929 19:06:54.757213 29399 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8af2c060feb946169db3fa38b38468ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8af2c060feb946169db3fa38b38468ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44115 } }
I20260929 19:06:54.757294 29399 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8af2c060feb946169db3fa38b38468ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:06:54.757331 29399 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8af2c060feb946169db3fa38b38468ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:06:54.757382 29399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8af2c060feb946169db3fa38b38468ce [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:06:54.757854 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 29331
I20260929 19:06:54.757987 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:06:54.757997 29399 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8af2c060feb946169db3fa38b38468ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8af2c060feb946169db3fa38b38468ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44115 } }
I20260929 19:06:54.758092 29399 leader_election.cc:302] T 00000000000000000000000000000000 P 8af2c060feb946169db3fa38b38468ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8af2c060feb946169db3fa38b38468ce; no voters:
I20260929 19:06:54.758190 8825 external_mini_cluster.cc:1428] Setting key 5457577c23a5d9162d865ac92a2ec66a
I20260929 19:06:54.758253 29399 leader_election.cc:288] T 00000000000000000000000000000000 P 8af2c060feb946169db3fa38b38468ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:06:54.758286 29402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8af2c060feb946169db3fa38b38468ce [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:06:54.758505 29402 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8af2c060feb946169db3fa38b38468ce [term 1 LEADER]: Becoming Leader. State: Replica: 8af2c060feb946169db3fa38b38468ce, State: Running, Role: LEADER
I20260929 19:06:54.758527 29399 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8af2c060feb946169db3fa38b38468ce [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:06:54.758718 29402 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8af2c060feb946169db3fa38b38468ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8af2c060feb946169db3fa38b38468ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44115 } }
I20260929 19:06:54.759102 29403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8af2c060feb946169db3fa38b38468ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8af2c060feb946169db3fa38b38468ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8af2c060feb946169db3fa38b38468ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44115 } } }
I20260929 19:06:54.759176 29403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8af2c060feb946169db3fa38b38468ce [sys.catalog]: This master's current role is: LEADER
I20260929 19:06:54.759315 29404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8af2c060feb946169db3fa38b38468ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8af2c060feb946169db3fa38b38468ce. Latest consensus state: current_term: 1 leader_uuid: "8af2c060feb946169db3fa38b38468ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8af2c060feb946169db3fa38b38468ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44115 } } }
I20260929 19:06:54.759366 29404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8af2c060feb946169db3fa38b38468ce [sys.catalog]: This master's current role is: LEADER
I20260929 19:06:54.759747 29408 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:06:54.760272 29408 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:06:54.761786 29408 catalog_manager.cc:1347] Generated new cluster ID: 5f5f28b86ede4f8f86c646550fbe59ce
I20260929 19:06:54.761843 29408 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 19:06:54 dist-test-slave-nmsz krb5kdc[29318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 19:06:54.769323 29406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:54.758649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41847 (local address 127.0.0.1:44115)
0929 19:06:54.758793 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:54.758802 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:54.759133 (+ 331us) server_negotiation.cc:407] Connection header received
0929 19:06:54.759259 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:54.759266 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:54.759318 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:54.759452 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:54.760399 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:54.761418 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:54.762492 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:54.763115 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:54.763227 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:54.765869 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:54.765873 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:54.765873 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:06:54.765883 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:54.767772 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:54.768172 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:54.768175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:54.768175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:54.768277 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:54.768492 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:54.768492 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:54.768493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:54.768752 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:54.768890 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:54.769022 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 19:06:54.769141 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":72,"threads_started":1}
I20260929 19:06:54.770419 8825 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:06:54.778512 29406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:54.771732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56889 (local address 127.0.0.1:44115)
0929 19:06:54.771769 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:54.771771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:54.771832 (+ 61us) server_negotiation.cc:407] Connection header received
0929 19:06:54.771899 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:54.771900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:54.771909 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:54.771963 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:54.772546 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:54.773225 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:54.774232 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:54.774554 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:54.774611 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:54.775902 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:54.775903 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:54.775903 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:06:54.775909 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:54.777388 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:54.777758 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:54.777759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:54.777759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:54.777808 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:54.778006 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:54.778006 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:54.778007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:54.778116 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:54.778194 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:54.778363 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 19:06:54.778430 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 19:06:54.780244 29408 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:06:54.780709 29408 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:06:54.785107 29408 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8af2c060feb946169db3fa38b38468ce: Generated new TSK 0
I20260929 19:06:54.785267 29408 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:06:54.788651 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 29331
2026-09-29T19:06:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1292 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708573736807-8825-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:06:54 dist-test-slave-nmsz krb5kdc[29430](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:06:54 dist-test-slave-nmsz krb5kdc[29430](info): set up 2 sockets
Sep 29 19:06:54 dist-test-slave-nmsz krb5kdc[29430](info): commencing operation
krb5kdc: starting...
W20260929 19:06:55.818250 8825 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Sep 29 19:06:55 dist-test-slave-nmsz krb5kdc[29430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:06:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:06:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708573736807-8825-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708573736807-8825-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 19:06:55.930014 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42007
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708573736807-8825-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:06:56.010074 29443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:06:56.010351 29443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:06:56.010398 29443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:06:56.012032 29443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:06:56.012089 29443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:06:56.012111 29443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:06:56.012130 29443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:06:56.012197 29443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:06:56.014017 29443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708573736807-8825-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:06:56.014278 29443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:06:56.014559 29443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:06:56.019104 29448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:56.019119 29449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:56.019181 29451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:56.019663 29443 server_base.cc:1034] running on GCE node
I20260929 19:06:56.019866 29443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:06:56.020092 29443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:06:56.021258 29443 hybrid_clock.cc:648] HybridClock initialized: now 1790708816021244 us; error 27 us; skew 500 ppm
Sep 29 19:06:56 dist-test-slave-nmsz krb5kdc[29430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:06:56.023718 29443 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 19:06:56.024134 29443 webserver.cc:460] Webserver started at http://127.0.0.1:45667/ using document root <none> and password file <none>
I20260929 19:06:56.024313 29443 fs_manager.cc:359] Metadata directory not provided
I20260929 19:06:56.024353 29443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:06:56.024434 29443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:06:56.025362 29443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "02b33f4c1a66411389ef7bfad109e183"
format_stamp: "Formatted at 2026-09-29 19:06:56 on dist-test-slave-nmsz"
server_key: "4b5cc2362c56c311875997e2686b4cda"
server_key_iv: "a8f16db405cc93f7ccafa22ac6dc5bb5"
server_key_version: "encryptionkey@0"
I20260929 19:06:56.025646 29443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "02b33f4c1a66411389ef7bfad109e183"
format_stamp: "Formatted at 2026-09-29 19:06:56 on dist-test-slave-nmsz"
server_key: "4b5cc2362c56c311875997e2686b4cda"
server_key_iv: "a8f16db405cc93f7ccafa22ac6dc5bb5"
server_key_version: "encryptionkey@0"
I20260929 19:06:56.026876 29443 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:06:56.027493 29458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:56.027642 29443 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:06:56.027719 29443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "02b33f4c1a66411389ef7bfad109e183"
format_stamp: "Formatted at 2026-09-29 19:06:56 on dist-test-slave-nmsz"
server_key: "4b5cc2362c56c311875997e2686b4cda"
server_key_iv: "a8f16db405cc93f7ccafa22ac6dc5bb5"
server_key_version: "encryptionkey@0"
I20260929 19:06:56.027770 29443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:06:56.044670 29443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:06:56.046694 29443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:06:56.046801 29443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:06:56.050414 29443 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42007
I20260929 19:06:56.050468 29510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42007 every 8 connection(s)
I20260929 19:06:56.050763 29443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:06:56.051646 29511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:06:56.053663 29511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02b33f4c1a66411389ef7bfad109e183: Bootstrap starting.
I20260929 19:06:56.054131 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 29443
I20260929 19:06:56.054227 29511 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02b33f4c1a66411389ef7bfad109e183: Neither blocks nor log segments found. Creating new log.
I20260929 19:06:56.054266 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:06:56.054476 8825 external_mini_cluster.cc:1428] Setting key 6176e81c067ce93bad73bdc8424166f0
I20260929 19:06:56.054564 29511 log.cc:824] T 00000000000000000000000000000000 P 02b33f4c1a66411389ef7bfad109e183: Log is configured to *not* fsync() on all Append() calls
I20260929 19:06:56.055313 29511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02b33f4c1a66411389ef7bfad109e183: No bootstrap required, opened a new log
I20260929 19:06:56.057406 29511 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02b33f4c1a66411389ef7bfad109e183 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02b33f4c1a66411389ef7bfad109e183" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42007 } }
I20260929 19:06:56.057595 29511 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02b33f4c1a66411389ef7bfad109e183 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:06:56.057643 29511 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02b33f4c1a66411389ef7bfad109e183 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02b33f4c1a66411389ef7bfad109e183, State: Initialized, Role: FOLLOWER
I20260929 19:06:56.057745 29511 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02b33f4c1a66411389ef7bfad109e183 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02b33f4c1a66411389ef7bfad109e183" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42007 } }
I20260929 19:06:56.057837 29511 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02b33f4c1a66411389ef7bfad109e183 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:06:56.057883 29511 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02b33f4c1a66411389ef7bfad109e183 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:06:56.057965 29511 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02b33f4c1a66411389ef7bfad109e183 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:06:56.058671 29511 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02b33f4c1a66411389ef7bfad109e183 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02b33f4c1a66411389ef7bfad109e183" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42007 } }
I20260929 19:06:56.058775 29511 leader_election.cc:302] T 00000000000000000000000000000000 P 02b33f4c1a66411389ef7bfad109e183 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02b33f4c1a66411389ef7bfad109e183; no voters:
I20260929 19:06:56.058959 29511 leader_election.cc:288] T 00000000000000000000000000000000 P 02b33f4c1a66411389ef7bfad109e183 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:06:56.058995 29516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02b33f4c1a66411389ef7bfad109e183 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:06:56.059197 29516 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02b33f4c1a66411389ef7bfad109e183 [term 1 LEADER]: Becoming Leader. State: Replica: 02b33f4c1a66411389ef7bfad109e183, State: Running, Role: LEADER
I20260929 19:06:56.059449 29516 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02b33f4c1a66411389ef7bfad109e183 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02b33f4c1a66411389ef7bfad109e183" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42007 } }
I20260929 19:06:56.059206 29511 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02b33f4c1a66411389ef7bfad109e183 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:06:56.060031 29515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:56.054953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55355 (local address 127.0.0.1:42007)
0929 19:06:56.055094 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:56.055101 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:56.055136 (+ 35us) server_negotiation.cc:407] Connection header received
0929 19:06:56.055602 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:56.055608 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:56.055652 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:56.055760 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:56.056019 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:56.056857 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:56.057981 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:56.058604 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:56.058724 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:56.059374 (+ 650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:56.059378 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:56.059378 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:06:56.059568 (+ 190us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:56.059604 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:06:56.059617 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:56.059679 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:56.059782 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 19:06:56.059843 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":75,"threads_started":1}
I20260929 19:06:56.060159 29518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02b33f4c1a66411389ef7bfad109e183 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02b33f4c1a66411389ef7bfad109e183" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02b33f4c1a66411389ef7bfad109e183" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42007 } } }
I20260929 19:06:56.060225 29518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02b33f4c1a66411389ef7bfad109e183 [sys.catalog]: This master's current role is: LEADER
I20260929 19:06:56.060468 29523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:06:56.060873 29517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02b33f4c1a66411389ef7bfad109e183 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02b33f4c1a66411389ef7bfad109e183. Latest consensus state: current_term: 1 leader_uuid: "02b33f4c1a66411389ef7bfad109e183" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02b33f4c1a66411389ef7bfad109e183" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42007 } } }
I20260929 19:06:56.060940 8825 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:06:56.060995 29517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02b33f4c1a66411389ef7bfad109e183 [sys.catalog]: This master's current role is: LEADER
I20260929 19:06:56.062247 29523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:06:56.063859 29523 catalog_manager.cc:1347] Generated new cluster ID: 098f43c3da1d43ca98ca900eb7f639de
I20260929 19:06:56.063903 29523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:06:56.066373 29515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:56.062391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40793 (local address 127.0.0.1:42007)
0929 19:06:56.062424 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:56.062426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:56.063050 (+ 624us) server_negotiation.cc:407] Connection header received
0929 19:06:56.063140 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:56.063141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:56.063149 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:56.063176 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:56.063475 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:56.064141 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:56.065141 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:56.065461 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:56.065519 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:56.066132 (+ 613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:56.066132 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:56.066133 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:06:56.066137 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:56.066149 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:06:56.066153 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:56.066199 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:56.066257 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 19:06:56.066274 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260929 19:06:56.070163 29523 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:06:56.070744 29523 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:06:56.079829 29523 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02b33f4c1a66411389ef7bfad109e183: Generated new TSK 0
I20260929 19:06:56.080027 29523 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:06:56.084546 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 29443
2026-09-29T19:06:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1296 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708573736807-8825-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:06:56 dist-test-slave-nmsz krb5kdc[29543](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:06:56 dist-test-slave-nmsz krb5kdc[29543](info): set up 2 sockets
Sep 29 19:06:56 dist-test-slave-nmsz krb5kdc[29543](info): commencing operation
krb5kdc: starting...
W20260929 19:06:57.114346 8825 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Sep 29 19:06:57 dist-test-slave-nmsz krb5kdc[29543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:06:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:06:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708573736807-8825-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708573736807-8825-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 19:06:57.224548 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45121
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708573736807-8825-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:06:57.301872 29556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:06:57.302086 29556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:06:57.302124 29556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:06:57.303244 29556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:06:57.303310 29556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:06:57.303337 29556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:06:57.303369 29556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:06:57.303393 29556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:06:57.304818 29556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708573736807-8825-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:06:57.305076 29556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:06:57.305284 29556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:06:57.310083 29562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:57.310073 29564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:57.310072 29561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:57.310175 29556 server_base.cc:1034] running on GCE node
I20260929 19:06:57.310480 29556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:06:57.310702 29556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:06:57.311873 29556 hybrid_clock.cc:648] HybridClock initialized: now 1790708817311888 us; error 56 us; skew 500 ppm
Sep 29 19:06:57 dist-test-slave-nmsz krb5kdc[29543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:06:57.319154 29556 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 19:06:57.319619 29556 webserver.cc:460] Webserver started at http://127.0.0.1:33829/ using document root <none> and password file <none>
I20260929 19:06:57.319835 29556 fs_manager.cc:359] Metadata directory not provided
I20260929 19:06:57.319903 29556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:06:57.320051 29556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:06:57.321048 29556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "53515ecf977e472b844a4ad3fee35938"
format_stamp: "Formatted at 2026-09-29 19:06:57 on dist-test-slave-nmsz"
server_key: "3edc4a0e6685f92f42abd9490e0e2256"
server_key_iv: "44b96b6a444b3175b7decce801a5e9f4"
server_key_version: "encryptionkey@0"
I20260929 19:06:57.321389 29556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "53515ecf977e472b844a4ad3fee35938"
format_stamp: "Formatted at 2026-09-29 19:06:57 on dist-test-slave-nmsz"
server_key: "3edc4a0e6685f92f42abd9490e0e2256"
server_key_iv: "44b96b6a444b3175b7decce801a5e9f4"
server_key_version: "encryptionkey@0"
I20260929 19:06:57.322708 29556 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260929 19:06:57.323433 29571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:57.323637 29556 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 19:06:57.323747 29556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "53515ecf977e472b844a4ad3fee35938"
format_stamp: "Formatted at 2026-09-29 19:06:57 on dist-test-slave-nmsz"
server_key: "3edc4a0e6685f92f42abd9490e0e2256"
server_key_iv: "44b96b6a444b3175b7decce801a5e9f4"
server_key_version: "encryptionkey@0"
I20260929 19:06:57.323822 29556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:06:57.327169 29556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:06:57.329341 29556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:06:57.329488 29556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:06:57.333118 29556 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45121
I20260929 19:06:57.333179 29623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45121 every 8 connection(s)
I20260929 19:06:57.333487 29556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:06:57.334465 29624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:06:57.336548 29624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53515ecf977e472b844a4ad3fee35938: Bootstrap starting.
I20260929 19:06:57.337087 29624 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53515ecf977e472b844a4ad3fee35938: Neither blocks nor log segments found. Creating new log.
I20260929 19:06:57.337363 29624 log.cc:824] T 00000000000000000000000000000000 P 53515ecf977e472b844a4ad3fee35938: Log is configured to *not* fsync() on all Append() calls
I20260929 19:06:57.338020 29624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53515ecf977e472b844a4ad3fee35938: No bootstrap required, opened a new log
I20260929 19:06:57.338558 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 29556
I20260929 19:06:57.338685 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:06:57.338899 8825 external_mini_cluster.cc:1428] Setting key 14f660244cafd3056881f3632424087c
I20260929 19:06:57.340024 29624 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53515ecf977e472b844a4ad3fee35938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53515ecf977e472b844a4ad3fee35938" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45121 } }
I20260929 19:06:57.340186 29624 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53515ecf977e472b844a4ad3fee35938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:06:57.340224 29624 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53515ecf977e472b844a4ad3fee35938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53515ecf977e472b844a4ad3fee35938, State: Initialized, Role: FOLLOWER
I20260929 19:06:57.340380 29624 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53515ecf977e472b844a4ad3fee35938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53515ecf977e472b844a4ad3fee35938" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45121 } }
I20260929 19:06:57.340502 29624 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53515ecf977e472b844a4ad3fee35938 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:06:57.340548 29624 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53515ecf977e472b844a4ad3fee35938 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:06:57.340621 29624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53515ecf977e472b844a4ad3fee35938 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:06:57.341321 29628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:57.339406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40077 (local address 127.0.0.1:45121)
0929 19:06:57.339597 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:57.339607 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:57.339639 (+ 32us) server_negotiation.cc:407] Connection header received
0929 19:06:57.339987 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:57.339996 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:57.340028 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:57.340177 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:57.340913 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:57.340916 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:57.340916 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:06:57.340920 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:57.340984 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:06:57.340995 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:57.341057 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:57.341159 (+ 102us) server_negotiation.cc:299] Negotiation successful
0929 19:06:57.341188 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":86,"threads_started":1}
I20260929 19:06:57.341729 29624 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53515ecf977e472b844a4ad3fee35938 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53515ecf977e472b844a4ad3fee35938" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45121 } }
I20260929 19:06:57.341849 29624 leader_election.cc:302] T 00000000000000000000000000000000 P 53515ecf977e472b844a4ad3fee35938 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53515ecf977e472b844a4ad3fee35938; no voters:
I20260929 19:06:57.342013 29624 leader_election.cc:288] T 00000000000000000000000000000000 P 53515ecf977e472b844a4ad3fee35938 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:06:57.342061 29629 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53515ecf977e472b844a4ad3fee35938 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:06:57.342244 29624 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53515ecf977e472b844a4ad3fee35938 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:06:57.342303 8825 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:06:57.342228 29629 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53515ecf977e472b844a4ad3fee35938 [term 1 LEADER]: Becoming Leader. State: Replica: 53515ecf977e472b844a4ad3fee35938, State: Running, Role: LEADER
I20260929 19:06:57.342473 29629 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53515ecf977e472b844a4ad3fee35938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53515ecf977e472b844a4ad3fee35938" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45121 } }
I20260929 19:06:57.342842 29631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53515ecf977e472b844a4ad3fee35938 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53515ecf977e472b844a4ad3fee35938. Latest consensus state: current_term: 1 leader_uuid: "53515ecf977e472b844a4ad3fee35938" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53515ecf977e472b844a4ad3fee35938" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45121 } } }
I20260929 19:06:57.343065 29631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53515ecf977e472b844a4ad3fee35938 [sys.catalog]: This master's current role is: LEADER
I20260929 19:06:57.343295 29638 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:06:57.343766 29638 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:06:57.343888 29630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53515ecf977e472b844a4ad3fee35938 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53515ecf977e472b844a4ad3fee35938" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53515ecf977e472b844a4ad3fee35938" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45121 } } }
I20260929 19:06:57.344208 29630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53515ecf977e472b844a4ad3fee35938 [sys.catalog]: This master's current role is: LEADER
I20260929 19:06:57.345496 29628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:57.344179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48779 (local address 127.0.0.1:45121)
0929 19:06:57.344208 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:57.344210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:57.344547 (+ 337us) server_negotiation.cc:407] Connection header received
0929 19:06:57.344627 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:57.344628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:57.344632 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:57.344686 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:57.345267 (+ 581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:57.345268 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:57.345269 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:06:57.345273 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:57.345284 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:06:57.345289 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:57.345340 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:57.345431 (+ 91us) server_negotiation.cc:299] Negotiation successful
0929 19:06:57.345444 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260929 19:06:57.345669 29638 catalog_manager.cc:1347] Generated new cluster ID: 38ef00f560aa4e3fa0819a1a12955749
I20260929 19:06:57.345711 29638 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:06:57.354249 29638 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:06:57.354410 29638 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:06:57.359544 29638 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53515ecf977e472b844a4ad3fee35938: Generated new TSK 0
I20260929 19:06:57.359715 29638 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:06:57.365803 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 29556
2026-09-29T19:06:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1281 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708573736807-8825-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:06:57 dist-test-slave-nmsz krb5kdc[29655](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:06:57 dist-test-slave-nmsz krb5kdc[29655](info): set up 2 sockets
Sep 29 19:06:57 dist-test-slave-nmsz krb5kdc[29655](info): commencing operation
krb5kdc: starting...
W20260929 19:06:58.386530 8825 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Sep 29 19:06:58 dist-test-slave-nmsz krb5kdc[29655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:06:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:06:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708573736807-8825-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708573736807-8825-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 19:06:58.497112 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41157
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708573736807-8825-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41157
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:06:58.574120 29668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:06:58.574355 29668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:06:58.574401 29668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:06:58.575531 29668 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:06:58.575596 29668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:06:58.575623 29668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:06:58.575655 29668 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:06:58.575680 29668 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:06:58.577098 29668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708573736807-8825-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:06:58.577347 29668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:06:58.577566 29668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:06:58.582079 29673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:58.582088 29676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:58.582091 29674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:58.582417 29668 server_base.cc:1034] running on GCE node
I20260929 19:06:58.582638 29668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:06:58.582906 29668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:06:58.584064 29668 hybrid_clock.cc:648] HybridClock initialized: now 1790708818584054 us; error 36 us; skew 500 ppm
Sep 29 19:06:58 dist-test-slave-nmsz krb5kdc[29655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:06:58.586454 29668 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 19:06:58.586932 29668 webserver.cc:460] Webserver started at http://127.0.0.1:38941/ using document root <none> and password file <none>
I20260929 19:06:58.587177 29668 fs_manager.cc:359] Metadata directory not provided
I20260929 19:06:58.587248 29668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:06:58.587378 29668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:06:58.588369 29668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "59317e7033d746fe86ef8dd51fc0ffc4"
format_stamp: "Formatted at 2026-09-29 19:06:58 on dist-test-slave-nmsz"
server_key: "c3de163675f84a9272787c3dfe4c80a8"
server_key_iv: "be6c25afab07d5f98062a5d9cf4de82f"
server_key_version: "encryptionkey@0"
I20260929 19:06:58.588724 29668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "59317e7033d746fe86ef8dd51fc0ffc4"
format_stamp: "Formatted at 2026-09-29 19:06:58 on dist-test-slave-nmsz"
server_key: "c3de163675f84a9272787c3dfe4c80a8"
server_key_iv: "be6c25afab07d5f98062a5d9cf4de82f"
server_key_version: "encryptionkey@0"
I20260929 19:06:58.589989 29668 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:06:58.590826 29683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:58.591044 29668 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 19:06:58.591140 29668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "59317e7033d746fe86ef8dd51fc0ffc4"
format_stamp: "Formatted at 2026-09-29 19:06:58 on dist-test-slave-nmsz"
server_key: "c3de163675f84a9272787c3dfe4c80a8"
server_key_iv: "be6c25afab07d5f98062a5d9cf4de82f"
server_key_version: "encryptionkey@0"
I20260929 19:06:58.591219 29668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:06:58.601753 29668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:06:58.603475 29668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:06:58.603632 29668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:06:58.607491 29668 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41157
I20260929 19:06:58.607528 29735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41157 every 8 connection(s)
I20260929 19:06:58.607827 29668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:06:58.608682 29736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:06:58.610802 29736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59317e7033d746fe86ef8dd51fc0ffc4: Bootstrap starting.
I20260929 19:06:58.611039 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 29668
I20260929 19:06:58.611143 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:06:58.611357 8825 external_mini_cluster.cc:1428] Setting key e9f43c1c5fd260b858525617d466aa82
I20260929 19:06:58.611497 29736 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59317e7033d746fe86ef8dd51fc0ffc4: Neither blocks nor log segments found. Creating new log.
I20260929 19:06:58.611877 29736 log.cc:824] T 00000000000000000000000000000000 P 59317e7033d746fe86ef8dd51fc0ffc4: Log is configured to *not* fsync() on all Append() calls
I20260929 19:06:58.612546 29736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59317e7033d746fe86ef8dd51fc0ffc4: No bootstrap required, opened a new log
I20260929 19:06:58.614818 29736 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59317e7033d746fe86ef8dd51fc0ffc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59317e7033d746fe86ef8dd51fc0ffc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41157 } }
I20260929 19:06:58.614971 29736 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59317e7033d746fe86ef8dd51fc0ffc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:06:58.615010 29736 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59317e7033d746fe86ef8dd51fc0ffc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59317e7033d746fe86ef8dd51fc0ffc4, State: Initialized, Role: FOLLOWER
I20260929 19:06:58.615114 29736 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59317e7033d746fe86ef8dd51fc0ffc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59317e7033d746fe86ef8dd51fc0ffc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41157 } }
I20260929 19:06:58.615178 29736 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59317e7033d746fe86ef8dd51fc0ffc4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:06:58.615355 29736 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59317e7033d746fe86ef8dd51fc0ffc4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:06:58.615432 29736 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59317e7033d746fe86ef8dd51fc0ffc4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:06:58.616050 29736 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59317e7033d746fe86ef8dd51fc0ffc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59317e7033d746fe86ef8dd51fc0ffc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41157 } }
I20260929 19:06:58.616158 29736 leader_election.cc:302] T 00000000000000000000000000000000 P 59317e7033d746fe86ef8dd51fc0ffc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59317e7033d746fe86ef8dd51fc0ffc4; no voters:
I20260929 19:06:58.616302 29736 leader_election.cc:288] T 00000000000000000000000000000000 P 59317e7033d746fe86ef8dd51fc0ffc4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:06:58.616356 29741 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59317e7033d746fe86ef8dd51fc0ffc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:06:58.616485 29741 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59317e7033d746fe86ef8dd51fc0ffc4 [term 1 LEADER]: Becoming Leader. State: Replica: 59317e7033d746fe86ef8dd51fc0ffc4, State: Running, Role: LEADER
I20260929 19:06:58.616580 29736 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59317e7033d746fe86ef8dd51fc0ffc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:06:58.616564 29741 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59317e7033d746fe86ef8dd51fc0ffc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59317e7033d746fe86ef8dd51fc0ffc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41157 } }
I20260929 19:06:58.616896 29743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59317e7033d746fe86ef8dd51fc0ffc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59317e7033d746fe86ef8dd51fc0ffc4. Latest consensus state: current_term: 1 leader_uuid: "59317e7033d746fe86ef8dd51fc0ffc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59317e7033d746fe86ef8dd51fc0ffc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41157 } } }
I20260929 19:06:58.616979 29743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59317e7033d746fe86ef8dd51fc0ffc4 [sys.catalog]: This master's current role is: LEADER
I20260929 19:06:58.617178 29746 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:06:58.617906 29746 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:06:58.618062 29742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59317e7033d746fe86ef8dd51fc0ffc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59317e7033d746fe86ef8dd51fc0ffc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59317e7033d746fe86ef8dd51fc0ffc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41157 } } }
I20260929 19:06:58.618136 29742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59317e7033d746fe86ef8dd51fc0ffc4 [sys.catalog]: This master's current role is: LEADER
Sep 29 19:06:58 dist-test-slave-nmsz krb5kdc[29655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 19:06:58.619551 29746 catalog_manager.cc:1347] Generated new cluster ID: 21b4ddac347746d5bdeb12d6584114b3
I20260929 19:06:58.619601 29746 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:06:58.623211 29739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:58.611843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.200:33775 (local address 127.0.0.1:41157)
0929 19:06:58.612013 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:58.612021 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:58.612054 (+ 33us) server_negotiation.cc:407] Connection header received
0929 19:06:58.612564 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:58.612571 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:58.612617 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:58.612770 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:58.613488 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:58.614355 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:58.615425 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:58.616036 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:58.616148 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:58.619254 (+ 3106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:58.619256 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:58.619257 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:06:58.619265 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:58.621186 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:58.621640 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:58.621642 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:58.621643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:58.621737 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:58.621980 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:58.621981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:58.621981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:58.622270 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:58.622482 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:58.622898 (+ 416us) server_negotiation.cc:299] Negotiation successful
0929 19:06:58.622997 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":88,"threads_started":1}
I20260929 19:06:58.624096 8825 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:06:58.628181 29746 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:06:58.628736 29746 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:06:58.635187 29739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:58.627976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.200:51579 (local address 127.0.0.1:41157)
0929 19:06:58.628011 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:58.628013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:58.628272 (+ 259us) server_negotiation.cc:407] Connection header received
0929 19:06:58.628369 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:58.628370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:58.628379 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:58.628441 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:58.629065 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:58.629750 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:58.630782 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:58.631094 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:58.631153 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:58.632488 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:58.632490 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:58.632490 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:06:58.632497 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:58.633982 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:58.634413 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:58.634414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:58.634415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:58.634470 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:06:58.634696 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:06:58.634696 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:06:58.634697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:06:58.634804 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:58.634892 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:58.635047 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 19:06:58.635106 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 19:06:58.642041 29746 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59317e7033d746fe86ef8dd51fc0ffc4: Generated new TSK 0
I20260929 19:06:58.642201 29746 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:06:58.644071 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 29668
W20260929 19:06:58.647812 29653 connection.cc:535] client connection to 127.0.0.1:41157 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41157): Connection reset by peer (error 104)
2026-09-29T19:06:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1277 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708573736807-8825-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:06:58 dist-test-slave-nmsz krb5kdc[29767](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:06:58 dist-test-slave-nmsz krb5kdc[29767](info): set up 2 sockets
Sep 29 19:06:58 dist-test-slave-nmsz krb5kdc[29767](info): commencing operation
krb5kdc: starting...
W20260929 19:06:59.674383 8825 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Sep 29 19:06:59 dist-test-slave-nmsz krb5kdc[29767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:06:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:06:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708573736807-8825-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708573736807-8825-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 19:06:59.788287 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44973
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708573736807-8825-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:06:59.865767 29780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:06:59.865960 29780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:06:59.865983 29780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:06:59.867046 29780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:06:59.867082 29780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:06:59.867096 29780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:06:59.867110 29780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:06:59.867121 29780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:06:59.868448 29780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708573736807-8825-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:06:59.868615 29780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:06:59.868763 29780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:06:59.873445 29788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:59.873427 29786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:06:59.873428 29785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:06:59.873456 29780 server_base.cc:1034] running on GCE node
I20260929 19:06:59.873876 29780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:06:59.874126 29780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:06:59.875265 29780 hybrid_clock.cc:648] HybridClock initialized: now 1790708819875246 us; error 35 us; skew 500 ppm
Sep 29 19:06:59 dist-test-slave-nmsz krb5kdc[29767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:06:59.877595 29780 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 19:06:59.878063 29780 webserver.cc:460] Webserver started at http://127.0.0.1:35579/ using document root <none> and password file <none>
I20260929 19:06:59.878276 29780 fs_manager.cc:359] Metadata directory not provided
I20260929 19:06:59.878376 29780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:06:59.878525 29780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:06:59.879508 29780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "1170aa4a663143189c27ee8dafbbc930"
format_stamp: "Formatted at 2026-09-29 19:06:59 on dist-test-slave-nmsz"
server_key: "b450956dae00b98e2769595ee3f9eed5"
server_key_iv: "e2ac48dc72b95b005fe0bca8d453aff3"
server_key_version: "encryptionkey@0"
I20260929 19:06:59.879839 29780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "1170aa4a663143189c27ee8dafbbc930"
format_stamp: "Formatted at 2026-09-29 19:06:59 on dist-test-slave-nmsz"
server_key: "b450956dae00b98e2769595ee3f9eed5"
server_key_iv: "e2ac48dc72b95b005fe0bca8d453aff3"
server_key_version: "encryptionkey@0"
I20260929 19:06:59.881086 29780 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 19:06:59.881769 29795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:06:59.881946 29780 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:06:59.882041 29780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "1170aa4a663143189c27ee8dafbbc930"
format_stamp: "Formatted at 2026-09-29 19:06:59 on dist-test-slave-nmsz"
server_key: "b450956dae00b98e2769595ee3f9eed5"
server_key_iv: "e2ac48dc72b95b005fe0bca8d453aff3"
server_key_version: "encryptionkey@0"
I20260929 19:06:59.882118 29780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:06:59.896750 29780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:06:59.898938 29780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:06:59.899092 29780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:06:59.903043 29780 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44973
I20260929 19:06:59.903093 29847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44973 every 8 connection(s)
I20260929 19:06:59.903398 29780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:06:59.904291 29848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:06:59.906308 29848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1170aa4a663143189c27ee8dafbbc930: Bootstrap starting.
I20260929 19:06:59.906831 29848 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1170aa4a663143189c27ee8dafbbc930: Neither blocks nor log segments found. Creating new log.
I20260929 19:06:59.907047 29848 log.cc:824] T 00000000000000000000000000000000 P 1170aa4a663143189c27ee8dafbbc930: Log is configured to *not* fsync() on all Append() calls
I20260929 19:06:59.907635 29848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1170aa4a663143189c27ee8dafbbc930: No bootstrap required, opened a new log
I20260929 19:06:59.909148 29848 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1170aa4a663143189c27ee8dafbbc930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1170aa4a663143189c27ee8dafbbc930" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44973 } }
I20260929 19:06:59.909271 29848 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1170aa4a663143189c27ee8dafbbc930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:06:59.909293 29848 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1170aa4a663143189c27ee8dafbbc930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1170aa4a663143189c27ee8dafbbc930, State: Initialized, Role: FOLLOWER
I20260929 19:06:59.909421 29848 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1170aa4a663143189c27ee8dafbbc930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1170aa4a663143189c27ee8dafbbc930" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44973 } }
I20260929 19:06:59.909502 29848 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1170aa4a663143189c27ee8dafbbc930 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:06:59.909524 29848 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1170aa4a663143189c27ee8dafbbc930 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:06:59.909550 29848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1170aa4a663143189c27ee8dafbbc930 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:06:59.909982 29848 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1170aa4a663143189c27ee8dafbbc930 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1170aa4a663143189c27ee8dafbbc930" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44973 } }
I20260929 19:06:59.910058 29848 leader_election.cc:302] T 00000000000000000000000000000000 P 1170aa4a663143189c27ee8dafbbc930 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1170aa4a663143189c27ee8dafbbc930; no voters:
I20260929 19:06:59.910171 29848 leader_election.cc:288] T 00000000000000000000000000000000 P 1170aa4a663143189c27ee8dafbbc930 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:06:59.910243 29851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1170aa4a663143189c27ee8dafbbc930 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:06:59.910403 29851 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1170aa4a663143189c27ee8dafbbc930 [term 1 LEADER]: Becoming Leader. State: Replica: 1170aa4a663143189c27ee8dafbbc930, State: Running, Role: LEADER
I20260929 19:06:59.910451 29848 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1170aa4a663143189c27ee8dafbbc930 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:06:59.910542 29851 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1170aa4a663143189c27ee8dafbbc930 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1170aa4a663143189c27ee8dafbbc930" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44973 } }
I20260929 19:06:59.910913 29853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1170aa4a663143189c27ee8dafbbc930 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1170aa4a663143189c27ee8dafbbc930. Latest consensus state: current_term: 1 leader_uuid: "1170aa4a663143189c27ee8dafbbc930" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1170aa4a663143189c27ee8dafbbc930" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44973 } } }
I20260929 19:06:59.911053 29853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1170aa4a663143189c27ee8dafbbc930 [sys.catalog]: This master's current role is: LEADER
I20260929 19:06:59.911127 29852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1170aa4a663143189c27ee8dafbbc930 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1170aa4a663143189c27ee8dafbbc930" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1170aa4a663143189c27ee8dafbbc930" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44973 } } }
I20260929 19:06:59.911293 29852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1170aa4a663143189c27ee8dafbbc930 [sys.catalog]: This master's current role is: LEADER
I20260929 19:06:59.911407 29860 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:06:59.911921 29860 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:06:59.912600 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 29780
I20260929 19:06:59.912707 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:06:59.912896 8825 external_mini_cluster.cc:1428] Setting key 9e7abf47842a93a40d437374c9d3c4ff
I20260929 19:06:59.913463 29860 catalog_manager.cc:1347] Generated new cluster ID: fb8cce01e3b54500828889b99cbe6268
I20260929 19:06:59.913515 29860 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:06:59.918561 29871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:59.913446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.200:46539 (local address 127.0.0.1:44973)
0929 19:06:59.914573 (+ 1127us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:59.914580 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:59.914601 (+ 21us) server_negotiation.cc:407] Connection header received
0929 19:06:59.914687 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:59.914692 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:59.914731 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:59.914857 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:59.915049 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:59.915929 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:59.916689 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:59.917068 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:59.917190 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:59.917831 (+ 641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:59.917835 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:59.917835 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:06:59.918005 (+ 170us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:59.918032 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:06:59.918042 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:59.918303 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:59.918386 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 19:06:59.918413 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1089,"thread_start_us":62,"threads_started":1}
I20260929 19:06:59.919478 8825 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:06:59.920789 29860 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:06:59.921445 29860 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:06:59.924275 29860 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1170aa4a663143189c27ee8dafbbc930: Generated new TSK 0
I20260929 19:06:59.924432 29860 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:06:59.927209 29871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:06:59.920727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.200:57265 (local address 127.0.0.1:44973)
0929 19:06:59.920758 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0929 19:06:59.920759 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:06:59.924041 (+ 3282us) server_negotiation.cc:407] Connection header received
0929 19:06:59.924089 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:06:59.924090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:06:59.924100 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:06:59.924167 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:06:59.924443 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:59.925235 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:59.925921 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:06:59.926238 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:06:59.926307 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:06:59.926982 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:06:59.926983 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:06:59.926983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:06:59.926988 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:06:59.927001 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:06:59.927005 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:06:59.927069 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 19:06:59.927128 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 19:06:59.927140 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260929 19:06:59.929060 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 29780
W20260929 19:06:59.932866 29765 connection.cc:535] client connection to 127.0.0.1:44973 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44973): Connection reset by peer (error 104)
2026-09-29T19:06:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1284 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708573736807-8825-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:06:59 dist-test-slave-nmsz krb5kdc[29879](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:06:59 dist-test-slave-nmsz krb5kdc[29879](info): set up 2 sockets
Sep 29 19:06:59 dist-test-slave-nmsz krb5kdc[29879](info): commencing operation
krb5kdc: starting...
W20260929 19:07:00.958462 8825 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Sep 29 19:07:01 dist-test-slave-nmsz krb5kdc[29879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:07:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:07:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708573736807-8825-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708573736807-8825-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 19:07:01.085522 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40181
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708573736807-8825-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:07:01.165591 29892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:07:01.165789 29892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:07:01.165812 29892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:07:01.167028 29892 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:07:01.167073 29892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:07:01.167088 29892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:07:01.167100 29892 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:07:01.167112 29892 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:07:01.168581 29892 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708573736807-8825-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:07:01.168814 29892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:07:01.169039 29892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 19:07:01.173765 29892 server_base.cc:1034] running on GCE node
W20260929 19:07:01.173722 29900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:07:01.173736 29897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:07:01.173739 29898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:07:01.174221 29892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:07:01.174542 29892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:07:01.175714 29892 hybrid_clock.cc:648] HybridClock initialized: now 1790708821175707 us; error 36 us; skew 500 ppm
Sep 29 19:07:01 dist-test-slave-nmsz krb5kdc[29879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:07:01.178069 29892 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 19:07:01.178499 29892 webserver.cc:460] Webserver started at http://127.0.0.1:41649/ using document root <none> and password file <none>
I20260929 19:07:01.178718 29892 fs_manager.cc:359] Metadata directory not provided
I20260929 19:07:01.178762 29892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:07:01.178926 29892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:07:01.179931 29892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "8685c368c5d14ac4aea8302d8ad6e48a"
format_stamp: "Formatted at 2026-09-29 19:07:01 on dist-test-slave-nmsz"
server_key: "ca3b9389b03d21612713e8926a394c7f"
server_key_iv: "f7943c11ffd3a5697409b945997d061a"
server_key_version: "encryptionkey@0"
I20260929 19:07:01.180290 29892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "8685c368c5d14ac4aea8302d8ad6e48a"
format_stamp: "Formatted at 2026-09-29 19:07:01 on dist-test-slave-nmsz"
server_key: "ca3b9389b03d21612713e8926a394c7f"
server_key_iv: "f7943c11ffd3a5697409b945997d061a"
server_key_version: "encryptionkey@0"
I20260929 19:07:01.181632 29892 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:07:01.182448 29907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:07:01.182624 29892 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:07:01.182722 29892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "8685c368c5d14ac4aea8302d8ad6e48a"
format_stamp: "Formatted at 2026-09-29 19:07:01 on dist-test-slave-nmsz"
server_key: "ca3b9389b03d21612713e8926a394c7f"
server_key_iv: "f7943c11ffd3a5697409b945997d061a"
server_key_version: "encryptionkey@0"
I20260929 19:07:01.182803 29892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:07:01.185865 29892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:07:01.188108 29892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:07:01.188262 29892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:07:01.191905 29892 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40181
I20260929 19:07:01.192001 29959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40181 every 8 connection(s)
I20260929 19:07:01.192287 29892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:07:01.193249 29960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:07:01.195544 29960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8685c368c5d14ac4aea8302d8ad6e48a: Bootstrap starting.
I20260929 19:07:01.196094 29960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8685c368c5d14ac4aea8302d8ad6e48a: Neither blocks nor log segments found. Creating new log.
I20260929 19:07:01.196375 29960 log.cc:824] T 00000000000000000000000000000000 P 8685c368c5d14ac4aea8302d8ad6e48a: Log is configured to *not* fsync() on all Append() calls
I20260929 19:07:01.197359 29960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8685c368c5d14ac4aea8302d8ad6e48a: No bootstrap required, opened a new log
I20260929 19:07:01.199018 29960 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8685c368c5d14ac4aea8302d8ad6e48a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8685c368c5d14ac4aea8302d8ad6e48a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40181 } }
I20260929 19:07:01.199148 29960 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8685c368c5d14ac4aea8302d8ad6e48a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:07:01.199170 29960 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8685c368c5d14ac4aea8302d8ad6e48a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8685c368c5d14ac4aea8302d8ad6e48a, State: Initialized, Role: FOLLOWER
I20260929 19:07:01.199325 29960 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8685c368c5d14ac4aea8302d8ad6e48a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8685c368c5d14ac4aea8302d8ad6e48a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40181 } }
I20260929 19:07:01.199386 29960 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8685c368c5d14ac4aea8302d8ad6e48a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:07:01.199426 29960 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8685c368c5d14ac4aea8302d8ad6e48a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:07:01.199504 29960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8685c368c5d14ac4aea8302d8ad6e48a [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:07:01.199656 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 29892
I20260929 19:07:01.199772 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:07:01.199957 8825 external_mini_cluster.cc:1428] Setting key e011b9a39a170b4b0d39c2b840136655
I20260929 19:07:01.200156 29960 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8685c368c5d14ac4aea8302d8ad6e48a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8685c368c5d14ac4aea8302d8ad6e48a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40181 } }
I20260929 19:07:01.200271 29960 leader_election.cc:302] T 00000000000000000000000000000000 P 8685c368c5d14ac4aea8302d8ad6e48a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8685c368c5d14ac4aea8302d8ad6e48a; no voters:
I20260929 19:07:01.200490 29960 leader_election.cc:288] T 00000000000000000000000000000000 P 8685c368c5d14ac4aea8302d8ad6e48a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:07:01.200510 29964 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8685c368c5d14ac4aea8302d8ad6e48a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:07:01.200690 29964 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8685c368c5d14ac4aea8302d8ad6e48a [term 1 LEADER]: Becoming Leader. State: Replica: 8685c368c5d14ac4aea8302d8ad6e48a, State: Running, Role: LEADER
I20260929 19:07:01.200773 29964 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8685c368c5d14ac4aea8302d8ad6e48a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8685c368c5d14ac4aea8302d8ad6e48a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40181 } }
I20260929 19:07:01.200788 29960 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8685c368c5d14ac4aea8302d8ad6e48a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:07:01.201148 29967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8685c368c5d14ac4aea8302d8ad6e48a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8685c368c5d14ac4aea8302d8ad6e48a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8685c368c5d14ac4aea8302d8ad6e48a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40181 } } }
I20260929 19:07:01.201243 29967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8685c368c5d14ac4aea8302d8ad6e48a [sys.catalog]: This master's current role is: LEADER
I20260929 19:07:01.201454 29966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8685c368c5d14ac4aea8302d8ad6e48a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8685c368c5d14ac4aea8302d8ad6e48a. Latest consensus state: current_term: 1 leader_uuid: "8685c368c5d14ac4aea8302d8ad6e48a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8685c368c5d14ac4aea8302d8ad6e48a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40181 } } }
I20260929 19:07:01.201562 29966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8685c368c5d14ac4aea8302d8ad6e48a [sys.catalog]: This master's current role is: LEADER
I20260929 19:07:01.201699 29969 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:07:01.202491 29969 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:07:01.202576 29965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:01.200420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.200:58811 (local address 127.0.0.1:40181)
0929 19:07:01.200604 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:01.200640 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:01.200677 (+ 37us) server_negotiation.cc:407] Connection header received
0929 19:07:01.200785 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:01.200791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:01.200821 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:01.200959 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:01.201954 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:01.201959 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:01.201959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:07:01.201964 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:01.202012 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:07:01.202029 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:01.202102 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:01.202306 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 19:07:01.202368 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":86,"threads_started":1}
I20260929 19:07:01.203516 8825 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:07:01.204288 29969 catalog_manager.cc:1347] Generated new cluster ID: 36f569b8e2da497a8eabf81f79288490
I20260929 19:07:01.204340 29969 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:07:01.206274 29965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:01.205095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.200:59009 (local address 127.0.0.1:40181)
0929 19:07:01.205129 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:01.205131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:01.205263 (+ 132us) server_negotiation.cc:407] Connection header received
0929 19:07:01.205333 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:01.205334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:01.205339 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:01.205400 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:01.206042 (+ 642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:01.206043 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:01.206044 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:07:01.206048 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:01.206062 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:07:01.206065 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:01.206124 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:01.206180 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 19:07:01.206194 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 19:07:01.224102 29969 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:07:01.224285 29969 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:07:01.227097 29969 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8685c368c5d14ac4aea8302d8ad6e48a: Generated new TSK 0
I20260929 19:07:01.227277 29969 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:07:01.231338 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 29892
2026-09-29T19:07:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1302 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708573736807-8825-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:07:01 dist-test-slave-nmsz krb5kdc[29991](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:07:01 dist-test-slave-nmsz krb5kdc[29991](info): set up 2 sockets
Sep 29 19:07:01 dist-test-slave-nmsz krb5kdc[29991](info): commencing operation
krb5kdc: starting...
W20260929 19:07:02.318485 8825 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Sep 29 19:07:02 dist-test-slave-nmsz krb5kdc[29991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:07:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:07:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708573736807-8825-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708573736807-8825-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 19:07:02.428443 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42011
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708573736807-8825-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42011
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:07:02.505105 30004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:07:02.505317 30004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:07:02.505355 30004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:07:02.506490 30004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:07:02.506556 30004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:07:02.506582 30004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:07:02.506615 30004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:07:02.506639 30004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:07:02.508052 30004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42011
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708573736807-8825-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:07:02.508306 30004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:07:02.508517 30004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:07:02.513064 30009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:07:02.513286 30012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:07:02.513303 30010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:07:02.513381 30004 server_base.cc:1034] running on GCE node
I20260929 19:07:02.513720 30004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:07:02.513983 30004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:07:02.515128 30004 hybrid_clock.cc:648] HybridClock initialized: now 1790708822515113 us; error 30 us; skew 500 ppm
Sep 29 19:07:02 dist-test-slave-nmsz krb5kdc[29991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:07:02.520884 30004 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 19:07:02.521317 30004 webserver.cc:460] Webserver started at http://127.0.0.1:34489/ using document root <none> and password file <none>
I20260929 19:07:02.521531 30004 fs_manager.cc:359] Metadata directory not provided
I20260929 19:07:02.521574 30004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:07:02.521706 30004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:07:02.522789 30004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "8ada35d204ac4523984fcc299a836aee"
format_stamp: "Formatted at 2026-09-29 19:07:02 on dist-test-slave-nmsz"
server_key: "e5847d6938358444aef94320e4a14d14"
server_key_iv: "76bdb30a1b9bfc985b8877c3d6aba02c"
server_key_version: "encryptionkey@0"
I20260929 19:07:02.523123 30004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "8ada35d204ac4523984fcc299a836aee"
format_stamp: "Formatted at 2026-09-29 19:07:02 on dist-test-slave-nmsz"
server_key: "e5847d6938358444aef94320e4a14d14"
server_key_iv: "76bdb30a1b9bfc985b8877c3d6aba02c"
server_key_version: "encryptionkey@0"
I20260929 19:07:02.524407 30004 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:07:02.525107 30019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:07:02.525290 30004 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:07:02.525388 30004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "8ada35d204ac4523984fcc299a836aee"
format_stamp: "Formatted at 2026-09-29 19:07:02 on dist-test-slave-nmsz"
server_key: "e5847d6938358444aef94320e4a14d14"
server_key_iv: "76bdb30a1b9bfc985b8877c3d6aba02c"
server_key_version: "encryptionkey@0"
I20260929 19:07:02.525455 30004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:07:02.533046 30004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:07:02.535200 30004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:07:02.535356 30004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:07:02.539008 30004 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42011
I20260929 19:07:02.539037 30071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42011 every 8 connection(s)
I20260929 19:07:02.539340 30004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:07:02.540202 30072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:07:02.542201 30072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ada35d204ac4523984fcc299a836aee: Bootstrap starting.
I20260929 19:07:02.542379 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 30004
I20260929 19:07:02.542498 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:07:02.542702 8825 external_mini_cluster.cc:1428] Setting key cfae5743121fae6e84d3690ace8b673e
I20260929 19:07:02.542924 30072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ada35d204ac4523984fcc299a836aee: Neither blocks nor log segments found. Creating new log.
I20260929 19:07:02.543259 30072 log.cc:824] T 00000000000000000000000000000000 P 8ada35d204ac4523984fcc299a836aee: Log is configured to *not* fsync() on all Append() calls
I20260929 19:07:02.543959 30072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ada35d204ac4523984fcc299a836aee: No bootstrap required, opened a new log
I20260929 19:07:02.546211 30072 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ada35d204ac4523984fcc299a836aee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ada35d204ac4523984fcc299a836aee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42011 } }
I20260929 19:07:02.546406 30072 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ada35d204ac4523984fcc299a836aee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:07:02.546473 30072 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ada35d204ac4523984fcc299a836aee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ada35d204ac4523984fcc299a836aee, State: Initialized, Role: FOLLOWER
I20260929 19:07:02.546594 30072 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ada35d204ac4523984fcc299a836aee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ada35d204ac4523984fcc299a836aee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42011 } }
I20260929 19:07:02.546663 30072 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ada35d204ac4523984fcc299a836aee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:07:02.546685 30072 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ada35d204ac4523984fcc299a836aee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:07:02.546751 30072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ada35d204ac4523984fcc299a836aee [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:07:02.547286 30072 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ada35d204ac4523984fcc299a836aee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ada35d204ac4523984fcc299a836aee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42011 } }
I20260929 19:07:02.547381 30072 leader_election.cc:302] T 00000000000000000000000000000000 P 8ada35d204ac4523984fcc299a836aee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ada35d204ac4523984fcc299a836aee; no voters:
I20260929 19:07:02.547528 30072 leader_election.cc:288] T 00000000000000000000000000000000 P 8ada35d204ac4523984fcc299a836aee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:07:02.547598 30077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ada35d204ac4523984fcc299a836aee [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:07:02.547753 30072 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ada35d204ac4523984fcc299a836aee [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:07:02.547778 30077 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ada35d204ac4523984fcc299a836aee [term 1 LEADER]: Becoming Leader. State: Replica: 8ada35d204ac4523984fcc299a836aee, State: Running, Role: LEADER
I20260929 19:07:02.547868 30077 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ada35d204ac4523984fcc299a836aee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ada35d204ac4523984fcc299a836aee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42011 } }
I20260929 19:07:02.548182 30077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ada35d204ac4523984fcc299a836aee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ada35d204ac4523984fcc299a836aee. Latest consensus state: current_term: 1 leader_uuid: "8ada35d204ac4523984fcc299a836aee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ada35d204ac4523984fcc299a836aee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42011 } } }
I20260929 19:07:02.548286 30078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ada35d204ac4523984fcc299a836aee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ada35d204ac4523984fcc299a836aee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ada35d204ac4523984fcc299a836aee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42011 } } }
I20260929 19:07:02.548309 30077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ada35d204ac4523984fcc299a836aee [sys.catalog]: This master's current role is: LEADER
I20260929 19:07:02.548394 30078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ada35d204ac4523984fcc299a836aee [sys.catalog]: This master's current role is: LEADER
I20260929 19:07:02.548556 30086 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:07:02.549024 30086 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 29 19:07:02 dist-test-slave-nmsz krb5kdc[29991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 19:07:02.551474 30086 catalog_manager.cc:1347] Generated new cluster ID: 1e39920ed7db4bdabed137dcb41e1bdf
I20260929 19:07:02.551524 30086 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:07:02.555088 30075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:02.543183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.200:37049 (local address 127.0.0.1:42011)
0929 19:07:02.543370 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:02.543379 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:02.543413 (+ 34us) server_negotiation.cc:407] Connection header received
0929 19:07:02.543969 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:02.543978 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:02.544038 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:02.544186 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:02.544755 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:02.546219 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:02.547239 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:02.547867 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:02.548028 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:02.551346 (+ 3318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:02.551350 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:02.551350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:07:02.551362 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:02.553261 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:02.553686 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:02.553689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:02.553689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:02.553789 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:02.554021 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:02.554022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:02.554022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:02.554338 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:02.554497 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:02.554761 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 19:07:02.554877 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":72,"threads_started":1}
I20260929 19:07:02.556135 8825 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:07:02.564291 30075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:02.557536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.200:41415 (local address 127.0.0.1:42011)
0929 19:07:02.557574 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:02.557576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:02.557727 (+ 151us) server_negotiation.cc:407] Connection header received
0929 19:07:02.557802 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:02.557803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:02.557811 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:02.557835 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:02.558477 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:02.559179 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:02.560112 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:02.560431 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:02.560465 (+ 34us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:02.561729 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:02.561730 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:02.561730 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:07:02.561736 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:02.563197 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:02.563578 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:02.563579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:02.563579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:02.563623 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:02.563806 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:02.563807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:02.563807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:02.563906 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:02.563969 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:02.564157 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 19:07:02.564209 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 19:07:02.568807 30086 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:07:02.569330 30086 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:07:02.573336 30086 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ada35d204ac4523984fcc299a836aee: Generated new TSK 0
I20260929 19:07:02.573493 30086 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:07:02.577659 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 30004
W20260929 19:07:02.581275 29989 connection.cc:535] client connection to 127.0.0.1:42011 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42011): Connection reset by peer (error 104)
2026-09-29T19:07:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1345 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708573736807-8825-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:07:02 dist-test-slave-nmsz krb5kdc[30103](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:07:02 dist-test-slave-nmsz krb5kdc[30103](info): set up 2 sockets
Sep 29 19:07:02 dist-test-slave-nmsz krb5kdc[30103](info): commencing operation
krb5kdc: starting...
W20260929 19:07:03.602209 8825 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Sep 29 19:07:03 dist-test-slave-nmsz krb5kdc[30103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:07:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:07:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708573736807-8825-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708573736807-8825-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 19:07:03.717901 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43509
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708573736807-8825-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:07:03.795212 30116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:07:03.795457 30116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:07:03.795495 30116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:07:03.796617 30116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:07:03.796693 30116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:07:03.796721 30116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:07:03.796752 30116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:07:03.796777 30116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:07:03.798243 30116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43509
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708573736807-8825-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:07:03.798534 30116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:07:03.798756 30116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:07:03.803921 30122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:07:03.803979 30116 server_base.cc:1034] running on GCE node
W20260929 19:07:03.803933 30124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:07:03.803936 30121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:07:03.804586 30116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:07:03.805011 30116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:07:03.806272 30116 hybrid_clock.cc:648] HybridClock initialized: now 1790708823806249 us; error 41 us; skew 500 ppm
Sep 29 19:07:03 dist-test-slave-nmsz krb5kdc[30103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:07:03.808825 30116 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 19:07:03.809297 30116 webserver.cc:460] Webserver started at http://127.0.0.1:33123/ using document root <none> and password file <none>
I20260929 19:07:03.809531 30116 fs_manager.cc:359] Metadata directory not provided
I20260929 19:07:03.809602 30116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:07:03.809754 30116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:07:03.810840 30116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "bedd9f60b50c42a5b70ea7614aa1b1dd"
format_stamp: "Formatted at 2026-09-29 19:07:03 on dist-test-slave-nmsz"
server_key: "fe981a982ba090f49dccbe10bca67ac2"
server_key_iv: "d5cc7454a4a79fd385ef1b3b0a7805be"
server_key_version: "encryptionkey@0"
I20260929 19:07:03.811189 30116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "bedd9f60b50c42a5b70ea7614aa1b1dd"
format_stamp: "Formatted at 2026-09-29 19:07:03 on dist-test-slave-nmsz"
server_key: "fe981a982ba090f49dccbe10bca67ac2"
server_key_iv: "d5cc7454a4a79fd385ef1b3b0a7805be"
server_key_version: "encryptionkey@0"
I20260929 19:07:03.812587 30116 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:07:03.813737 30131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:07:03.813944 30116 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:07:03.814060 30116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "bedd9f60b50c42a5b70ea7614aa1b1dd"
format_stamp: "Formatted at 2026-09-29 19:07:03 on dist-test-slave-nmsz"
server_key: "fe981a982ba090f49dccbe10bca67ac2"
server_key_iv: "d5cc7454a4a79fd385ef1b3b0a7805be"
server_key_version: "encryptionkey@0"
I20260929 19:07:03.814143 30116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:07:03.832183 30116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:07:03.834499 30116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:07:03.834663 30116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:07:03.838438 30116 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43509
I20260929 19:07:03.838488 30183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43509 every 8 connection(s)
I20260929 19:07:03.838869 30116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:07:03.839663 30184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:07:03.841744 30184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bedd9f60b50c42a5b70ea7614aa1b1dd: Bootstrap starting.
I20260929 19:07:03.842123 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 30116
I20260929 19:07:03.842252 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:07:03.842269 30184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bedd9f60b50c42a5b70ea7614aa1b1dd: Neither blocks nor log segments found. Creating new log.
I20260929 19:07:03.842489 8825 external_mini_cluster.cc:1428] Setting key d4b230b2018abadeb7e6943a968c50e8
I20260929 19:07:03.842586 30184 log.cc:824] T 00000000000000000000000000000000 P bedd9f60b50c42a5b70ea7614aa1b1dd: Log is configured to *not* fsync() on all Append() calls
I20260929 19:07:03.843366 30184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bedd9f60b50c42a5b70ea7614aa1b1dd: No bootstrap required, opened a new log
I20260929 19:07:03.845227 30184 raft_consensus.cc:348] T 00000000000000000000000000000000 P bedd9f60b50c42a5b70ea7614aa1b1dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bedd9f60b50c42a5b70ea7614aa1b1dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43509 } }
I20260929 19:07:03.845382 30184 raft_consensus.cc:374] T 00000000000000000000000000000000 P bedd9f60b50c42a5b70ea7614aa1b1dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:07:03.845429 30184 raft_consensus.cc:729] T 00000000000000000000000000000000 P bedd9f60b50c42a5b70ea7614aa1b1dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bedd9f60b50c42a5b70ea7614aa1b1dd, State: Initialized, Role: FOLLOWER
I20260929 19:07:03.845606 30184 consensus_queue.cc:260] T 00000000000000000000000000000000 P bedd9f60b50c42a5b70ea7614aa1b1dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bedd9f60b50c42a5b70ea7614aa1b1dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43509 } }
I20260929 19:07:03.845700 30184 raft_consensus.cc:388] T 00000000000000000000000000000000 P bedd9f60b50c42a5b70ea7614aa1b1dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:07:03.845744 30184 raft_consensus.cc:482] T 00000000000000000000000000000000 P bedd9f60b50c42a5b70ea7614aa1b1dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:07:03.845791 30184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bedd9f60b50c42a5b70ea7614aa1b1dd [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:07:03.846304 30184 raft_consensus.cc:504] T 00000000000000000000000000000000 P bedd9f60b50c42a5b70ea7614aa1b1dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bedd9f60b50c42a5b70ea7614aa1b1dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43509 } }
I20260929 19:07:03.846462 30184 leader_election.cc:302] T 00000000000000000000000000000000 P bedd9f60b50c42a5b70ea7614aa1b1dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bedd9f60b50c42a5b70ea7614aa1b1dd; no voters:
I20260929 19:07:03.846614 30184 leader_election.cc:288] T 00000000000000000000000000000000 P bedd9f60b50c42a5b70ea7614aa1b1dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:07:03.846635 30189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bedd9f60b50c42a5b70ea7614aa1b1dd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:07:03.846827 30189 raft_consensus.cc:686] T 00000000000000000000000000000000 P bedd9f60b50c42a5b70ea7614aa1b1dd [term 1 LEADER]: Becoming Leader. State: Replica: bedd9f60b50c42a5b70ea7614aa1b1dd, State: Running, Role: LEADER
I20260929 19:07:03.846916 30184 sys_catalog.cc:564] T 00000000000000000000000000000000 P bedd9f60b50c42a5b70ea7614aa1b1dd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:07:03.846966 30189 consensus_queue.cc:237] T 00000000000000000000000000000000 P bedd9f60b50c42a5b70ea7614aa1b1dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bedd9f60b50c42a5b70ea7614aa1b1dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43509 } }
I20260929 19:07:03.847886 30188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:03.843005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.200:59847 (local address 127.0.0.1:43509)
0929 19:07:03.843186 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:03.843195 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:03.843224 (+ 29us) server_negotiation.cc:407] Connection header received
0929 19:07:03.843699 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:03.843705 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:03.843752 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:03.843893 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:03.844114 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:03.844968 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:03.845988 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:03.846468 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:03.846581 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:03.847216 (+ 635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:03.847219 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:03.847220 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:07:03.847427 (+ 207us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:03.847463 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:07:03.847479 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:03.847571 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:03.847661 (+ 90us) server_negotiation.cc:299] Negotiation successful
0929 19:07:03.847687 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":79,"threads_started":1}
I20260929 19:07:03.848349 30190 sys_catalog.cc:455] T 00000000000000000000000000000000 P bedd9f60b50c42a5b70ea7614aa1b1dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader bedd9f60b50c42a5b70ea7614aa1b1dd. Latest consensus state: current_term: 1 leader_uuid: "bedd9f60b50c42a5b70ea7614aa1b1dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bedd9f60b50c42a5b70ea7614aa1b1dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43509 } } }
I20260929 19:07:03.848436 30190 sys_catalog.cc:458] T 00000000000000000000000000000000 P bedd9f60b50c42a5b70ea7614aa1b1dd [sys.catalog]: This master's current role is: LEADER
I20260929 19:07:03.848701 30191 sys_catalog.cc:455] T 00000000000000000000000000000000 P bedd9f60b50c42a5b70ea7614aa1b1dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bedd9f60b50c42a5b70ea7614aa1b1dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bedd9f60b50c42a5b70ea7614aa1b1dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43509 } } }
I20260929 19:07:03.848941 8825 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:07:03.848981 30191 sys_catalog.cc:458] T 00000000000000000000000000000000 P bedd9f60b50c42a5b70ea7614aa1b1dd [sys.catalog]: This master's current role is: LEADER
W20260929 19:07:03.849227 30205 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bedd9f60b50c42a5b70ea7614aa1b1dd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 19:07:03.849285 30205 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:07:03.849329 30204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:07:03.849479 30204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:07:03.851042 30204 catalog_manager.cc:1347] Generated new cluster ID: d0bd5adfefac47188e4d32b71511a0f4
I20260929 19:07:03.851099 30204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:07:03.853407 30188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:03.850341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.200:60739 (local address 127.0.0.1:43509)
0929 19:07:03.850378 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:03.850379 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:03.850624 (+ 245us) server_negotiation.cc:407] Connection header received
0929 19:07:03.850702 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:03.850703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:03.850713 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:03.850768 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:03.851003 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:03.851512 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:03.852216 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:03.852444 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:03.852529 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:03.853163 (+ 634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:03.853165 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:03.853165 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:07:03.853169 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:03.853182 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:07:03.853186 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:03.853238 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:03.853297 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 19:07:03.853306 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 19:07:03.859732 30204 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:07:03.860417 30204 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:07:03.866809 30204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bedd9f60b50c42a5b70ea7614aa1b1dd: Generated new TSK 0
I20260929 19:07:03.866967 30204 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:07:03.869694 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 30116
W20260929 19:07:03.873857 30101 connection.cc:535] client connection to 127.0.0.1:43509 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43509): Connection reset by peer (error 104)
2026-09-29T19:07:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1292 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708573736807-8825-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:07:03 dist-test-slave-nmsz krb5kdc[30215](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:07:03 dist-test-slave-nmsz krb5kdc[30215](info): set up 2 sockets
Sep 29 19:07:03 dist-test-slave-nmsz krb5kdc[30215](info): commencing operation
krb5kdc: starting...
W20260929 19:07:04.978624 8825 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Sep 29 19:07:05 dist-test-slave-nmsz krb5kdc[30215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:07:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:07:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708573736807-8825-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708573736807-8825-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 19:07:05.090413 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35105
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708573736807-8825-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:07:05.190025 30228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:07:05.190220 30228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:07:05.190245 30228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:07:05.191447 30228 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:07:05.191488 30228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:07:05.191504 30228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:07:05.191519 30228 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:07:05.191531 30228 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:07:05.193037 30228 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35105
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708573736807-8825-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:07:05.193248 30228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:07:05.193536 30228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:07:05.198400 30236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:07:05.198458 30234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:07:05.198419 30233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:07:05.198596 30228 server_base.cc:1034] running on GCE node
I20260929 19:07:05.198834 30228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:07:05.199115 30228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:07:05.200310 30228 hybrid_clock.cc:648] HybridClock initialized: now 1790708825200304 us; error 38 us; skew 500 ppm
Sep 29 19:07:05 dist-test-slave-nmsz krb5kdc[30215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:07:05.202842 30228 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 19:07:05.203294 30228 webserver.cc:460] Webserver started at http://127.0.0.1:35505/ using document root <none> and password file <none>
I20260929 19:07:05.203528 30228 fs_manager.cc:359] Metadata directory not provided
I20260929 19:07:05.203606 30228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:07:05.203748 30228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:07:05.204766 30228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "8a6a5d616a1b4c5b821a4963d02f4d7d"
format_stamp: "Formatted at 2026-09-29 19:07:05 on dist-test-slave-nmsz"
server_key: "75bb4af2e71e79acbf9ff15c9d77369f"
server_key_iv: "6a6ca6abde366d42f9517987f635875a"
server_key_version: "encryptionkey@0"
I20260929 19:07:05.205117 30228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "8a6a5d616a1b4c5b821a4963d02f4d7d"
format_stamp: "Formatted at 2026-09-29 19:07:05 on dist-test-slave-nmsz"
server_key: "75bb4af2e71e79acbf9ff15c9d77369f"
server_key_iv: "6a6ca6abde366d42f9517987f635875a"
server_key_version: "encryptionkey@0"
I20260929 19:07:05.206465 30228 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:07:05.207192 30243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:07:05.207356 30228 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:07:05.207460 30228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "8a6a5d616a1b4c5b821a4963d02f4d7d"
format_stamp: "Formatted at 2026-09-29 19:07:05 on dist-test-slave-nmsz"
server_key: "75bb4af2e71e79acbf9ff15c9d77369f"
server_key_iv: "6a6ca6abde366d42f9517987f635875a"
server_key_version: "encryptionkey@0"
I20260929 19:07:05.207543 30228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:07:05.210623 30228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:07:05.212464 30228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:07:05.212611 30228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:07:05.216554 30228 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35105
I20260929 19:07:05.216583 30295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35105 every 8 connection(s)
I20260929 19:07:05.216890 30228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:07:05.217769 30296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:07:05.219875 30296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a6a5d616a1b4c5b821a4963d02f4d7d: Bootstrap starting.
I20260929 19:07:05.220399 30296 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a6a5d616a1b4c5b821a4963d02f4d7d: Neither blocks nor log segments found. Creating new log.
I20260929 19:07:05.220664 30296 log.cc:824] T 00000000000000000000000000000000 P 8a6a5d616a1b4c5b821a4963d02f4d7d: Log is configured to *not* fsync() on all Append() calls
I20260929 19:07:05.221293 30296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a6a5d616a1b4c5b821a4963d02f4d7d: No bootstrap required, opened a new log
I20260929 19:07:05.223181 30296 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a6a5d616a1b4c5b821a4963d02f4d7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a6a5d616a1b4c5b821a4963d02f4d7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35105 } }
I20260929 19:07:05.223308 30296 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a6a5d616a1b4c5b821a4963d02f4d7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:07:05.223330 30296 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a6a5d616a1b4c5b821a4963d02f4d7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a6a5d616a1b4c5b821a4963d02f4d7d, State: Initialized, Role: FOLLOWER
I20260929 19:07:05.223470 30296 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a6a5d616a1b4c5b821a4963d02f4d7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a6a5d616a1b4c5b821a4963d02f4d7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35105 } }
I20260929 19:07:05.223532 30296 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a6a5d616a1b4c5b821a4963d02f4d7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:07:05.223552 30296 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a6a5d616a1b4c5b821a4963d02f4d7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:07:05.223613 30296 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a6a5d616a1b4c5b821a4963d02f4d7d [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:07:05.224159 30296 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a6a5d616a1b4c5b821a4963d02f4d7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a6a5d616a1b4c5b821a4963d02f4d7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35105 } }
I20260929 19:07:05.224265 30296 leader_election.cc:302] T 00000000000000000000000000000000 P 8a6a5d616a1b4c5b821a4963d02f4d7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a6a5d616a1b4c5b821a4963d02f4d7d; no voters:
I20260929 19:07:05.224434 30296 leader_election.cc:288] T 00000000000000000000000000000000 P 8a6a5d616a1b4c5b821a4963d02f4d7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:07:05.224493 30299 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a6a5d616a1b4c5b821a4963d02f4d7d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:07:05.224570 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 30228
I20260929 19:07:05.224646 30299 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a6a5d616a1b4c5b821a4963d02f4d7d [term 1 LEADER]: Becoming Leader. State: Replica: 8a6a5d616a1b4c5b821a4963d02f4d7d, State: Running, Role: LEADER
I20260929 19:07:05.224690 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:07:05.224686 30296 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a6a5d616a1b4c5b821a4963d02f4d7d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:07:05.224728 30299 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a6a5d616a1b4c5b821a4963d02f4d7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a6a5d616a1b4c5b821a4963d02f4d7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35105 } }
I20260929 19:07:05.224881 8825 external_mini_cluster.cc:1428] Setting key 5f9160d8cd34538695b5db76b75d1cb5
I20260929 19:07:05.225090 30299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a6a5d616a1b4c5b821a4963d02f4d7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a6a5d616a1b4c5b821a4963d02f4d7d. Latest consensus state: current_term: 1 leader_uuid: "8a6a5d616a1b4c5b821a4963d02f4d7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a6a5d616a1b4c5b821a4963d02f4d7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35105 } } }
I20260929 19:07:05.225138 30300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a6a5d616a1b4c5b821a4963d02f4d7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a6a5d616a1b4c5b821a4963d02f4d7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a6a5d616a1b4c5b821a4963d02f4d7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35105 } } }
I20260929 19:07:05.225248 30300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a6a5d616a1b4c5b821a4963d02f4d7d [sys.catalog]: This master's current role is: LEADER
I20260929 19:07:05.225443 30299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a6a5d616a1b4c5b821a4963d02f4d7d [sys.catalog]: This master's current role is: LEADER
I20260929 19:07:05.225605 30304 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:07:05.226158 30304 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:07:05.227689 30309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:05.225515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.200:32971 (local address 127.0.0.1:35105)
0929 19:07:05.225982 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:05.225990 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:05.226019 (+ 29us) server_negotiation.cc:407] Connection header received
0929 19:07:05.226126 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:05.226132 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:05.226161 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:05.226605 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:05.227036 (+ 431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:05.227039 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:05.227039 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:07:05.227042 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:05.227075 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:07:05.227087 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:05.227155 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:05.227327 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 19:07:05.227396 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":91,"threads_started":1}
I20260929 19:07:05.228149 30304 catalog_manager.cc:1347] Generated new cluster ID: e5198fe7474d44128fb1c34bbe6b0cd7
I20260929 19:07:05.228197 30304 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:07:05.228587 8825 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:07:05.231073 30309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:05.229969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.200:39169 (local address 127.0.0.1:35105)
0929 19:07:05.230003 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:05.230005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:05.230139 (+ 134us) server_negotiation.cc:407] Connection header received
0929 19:07:05.230203 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:05.230204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:05.230209 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:05.230258 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:05.230848 (+ 590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:05.230850 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:05.230850 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:07:05.230854 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:05.230866 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:07:05.230870 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:05.230932 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:05.230980 (+ 48us) server_negotiation.cc:299] Negotiation successful
0929 19:07:05.230994 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 19:07:05.235095 30304 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:07:05.235239 30304 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:07:05.246614 30304 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a6a5d616a1b4c5b821a4963d02f4d7d: Generated new TSK 0
I20260929 19:07:05.246775 30304 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:07:05.254684 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 30228
2026-09-29T19:07:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1384 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708573736807-8825-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:07:05 dist-test-slave-nmsz krb5kdc[30327](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:07:05 dist-test-slave-nmsz krb5kdc[30327](info): set up 2 sockets
Sep 29 19:07:05 dist-test-slave-nmsz krb5kdc[30327](info): commencing operation
krb5kdc: starting...
W20260929 19:07:06.282519 8825 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Sep 29 19:07:06 dist-test-slave-nmsz krb5kdc[30327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:07:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:07:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.158.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
I20260929 19:07:06.397120 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:33045
--webserver_interface=127.8.158.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:34359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:33045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:07:06.474996 30340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:07:06.475207 30340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:07:06.475241 30340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:07:06.476339 30340 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:07:06.476404 30340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:07:06.476429 30340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:07:06.476459 30340 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:07:06.476490 30340 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:07:06.477900 30340 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:34359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:33045
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:33045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.158.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:07:06.478158 30340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:07:06.478428 30340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:07:06.483057 30346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:07:06.483088 30345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:07:06.483069 30348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:07:06.483316 30340 server_base.cc:1034] running on GCE node
I20260929 19:07:06.483536 30340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:07:06.483803 30340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:07:06.484980 30340 hybrid_clock.cc:648] HybridClock initialized: now 1790708826484977 us; error 38 us; skew 500 ppm
Sep 29 19:07:06 dist-test-slave-nmsz krb5kdc[30327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708826, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:07:06.487504 30340 init.cc:377] Logged in from keytab as kudu/127.8.158.126@KRBTEST.COM (short username kudu)
I20260929 19:07:06.487864 30340 webserver.cc:460] Webserver started at http://127.8.158.126:45655/ using document root <none> and password file <none>
I20260929 19:07:06.488044 30340 fs_manager.cc:359] Metadata directory not provided
I20260929 19:07:06.488083 30340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:07:06.488165 30340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:07:06.489080 30340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "64c1bddb485245659752d98a07fa0bc8"
format_stamp: "Formatted at 2026-09-29 19:07:06 on dist-test-slave-nmsz"
server_key: "d26cf79c9313989ade96a44351f9748f"
server_key_iv: "17e4689427cf075782a9277d289a07df"
server_key_version: "encryptionkey@0"
I20260929 19:07:06.489398 30340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "64c1bddb485245659752d98a07fa0bc8"
format_stamp: "Formatted at 2026-09-29 19:07:06 on dist-test-slave-nmsz"
server_key: "d26cf79c9313989ade96a44351f9748f"
server_key_iv: "17e4689427cf075782a9277d289a07df"
server_key_version: "encryptionkey@0"
I20260929 19:07:06.490666 30340 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:07:06.491364 30355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:07:06.491530 30340 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:07:06.491600 30340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "64c1bddb485245659752d98a07fa0bc8"
format_stamp: "Formatted at 2026-09-29 19:07:06 on dist-test-slave-nmsz"
server_key: "d26cf79c9313989ade96a44351f9748f"
server_key_iv: "17e4689427cf075782a9277d289a07df"
server_key_version: "encryptionkey@0"
I20260929 19:07:06.491649 30340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:07:06.522428 30340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:07:06.524463 30340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:07:06.524573 30340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:07:06.528306 30340 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.126:33045
I20260929 19:07:06.528378 30407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.126:33045 every 8 connection(s)
I20260929 19:07:06.528685 30340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:07:06.529544 30408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:07:06.531337 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 30340
I20260929 19:07:06.531472 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:07:06.531666 8825 external_mini_cluster.cc:1428] Setting key f846ddb6b939b2b0f4bc8e697bd35ea5
I20260929 19:07:06.531996 30408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64c1bddb485245659752d98a07fa0bc8: Bootstrap starting.
I20260929 19:07:06.532588 30408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64c1bddb485245659752d98a07fa0bc8: Neither blocks nor log segments found. Creating new log.
I20260929 19:07:06.532861 30408 log.cc:824] T 00000000000000000000000000000000 P 64c1bddb485245659752d98a07fa0bc8: Log is configured to *not* fsync() on all Append() calls
I20260929 19:07:06.533666 30408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64c1bddb485245659752d98a07fa0bc8: No bootstrap required, opened a new log
I20260929 19:07:06.535504 30408 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64c1bddb485245659752d98a07fa0bc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64c1bddb485245659752d98a07fa0bc8" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 33045 } }
I20260929 19:07:06.535657 30408 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64c1bddb485245659752d98a07fa0bc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:07:06.535704 30408 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64c1bddb485245659752d98a07fa0bc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64c1bddb485245659752d98a07fa0bc8, State: Initialized, Role: FOLLOWER
I20260929 19:07:06.535835 30408 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64c1bddb485245659752d98a07fa0bc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64c1bddb485245659752d98a07fa0bc8" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 33045 } }
I20260929 19:07:06.535933 30408 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64c1bddb485245659752d98a07fa0bc8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:07:06.535990 30408 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64c1bddb485245659752d98a07fa0bc8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:07:06.536041 30408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64c1bddb485245659752d98a07fa0bc8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:07:06.536691 30408 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64c1bddb485245659752d98a07fa0bc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64c1bddb485245659752d98a07fa0bc8" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 33045 } }
I20260929 19:07:06.536810 30408 leader_election.cc:302] T 00000000000000000000000000000000 P 64c1bddb485245659752d98a07fa0bc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64c1bddb485245659752d98a07fa0bc8; no voters:
I20260929 19:07:06.537016 30408 leader_election.cc:288] T 00000000000000000000000000000000 P 64c1bddb485245659752d98a07fa0bc8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:07:06.537035 30413 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64c1bddb485245659752d98a07fa0bc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:07:06.537145 30413 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64c1bddb485245659752d98a07fa0bc8 [term 1 LEADER]: Becoming Leader. State: Replica: 64c1bddb485245659752d98a07fa0bc8, State: Running, Role: LEADER
I20260929 19:07:06.537271 30408 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64c1bddb485245659752d98a07fa0bc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:07:06.537298 30413 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64c1bddb485245659752d98a07fa0bc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64c1bddb485245659752d98a07fa0bc8" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 33045 } }
I20260929 19:07:06.537611 30413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64c1bddb485245659752d98a07fa0bc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64c1bddb485245659752d98a07fa0bc8. Latest consensus state: current_term: 1 leader_uuid: "64c1bddb485245659752d98a07fa0bc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64c1bddb485245659752d98a07fa0bc8" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 33045 } } }
I20260929 19:07:06.537624 30414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64c1bddb485245659752d98a07fa0bc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64c1bddb485245659752d98a07fa0bc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64c1bddb485245659752d98a07fa0bc8" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 33045 } } }
I20260929 19:07:06.537706 30413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64c1bddb485245659752d98a07fa0bc8 [sys.catalog]: This master's current role is: LEADER
I20260929 19:07:06.537721 30414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64c1bddb485245659752d98a07fa0bc8 [sys.catalog]: This master's current role is: LEADER
Sep 29 19:07:06 dist-test-slave-nmsz krb5kdc[30327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:07:06.538450 30420 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:07:06.538965 30420 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:07:06.540988 30420 catalog_manager.cc:1347] Generated new cluster ID: 6a279ef77054443ba2138fde6c5ad73c
I20260929 19:07:06.541033 30420 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:07:06.542229 30411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:06.532158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33783 (local address 127.8.158.126:33045)
0929 19:07:06.532334 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:06.532342 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:06.532383 (+ 41us) server_negotiation.cc:407] Connection header received
0929 19:07:06.532839 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:06.532845 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:06.532890 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:06.533068 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:06.533769 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:06.534678 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:06.535395 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:06.535882 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:06.535995 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:06.538679 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:06.538683 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:06.538683 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:07:06.538695 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:06.540749 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:06.541129 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:06.541132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:06.541132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:06.541224 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:06.541459 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:06.541460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:06.541460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:06.541681 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:06.541859 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:06.541966 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 19:07:06.542061 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
I20260929 19:07:06.543188 8825 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:07:06.546993 30420 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:07:06.547489 30420 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:07:06.555552 30420 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64c1bddb485245659752d98a07fa0bc8: Generated new TSK 0
I20260929 19:07:06.555729 30420 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:07:06.557548 30411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:06.546651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53683 (local address 127.8.158.126:33045)
0929 19:07:06.546688 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:06.546690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:06.550483 (+ 3793us) server_negotiation.cc:407] Connection header received
0929 19:07:06.550511 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:06.550511 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:06.550522 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:06.550584 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:06.551196 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:06.551982 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:06.553003 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:06.553297 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:06.553364 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:06.554725 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:06.554726 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:06.554727 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:07:06.554733 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:06.556418 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:06.556806 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:06.556807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:06.556807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:06.556867 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:06.557078 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:06.557079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:06.557079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:06.557200 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:06.557308 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:06.557399 (+ 91us) server_negotiation.cc:299] Negotiation successful
0929 19:07:06.557460 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 19:07:06.559671 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 30340
W20260929 19:07:06.563748 30325 connection.cc:535] client connection to 127.8.158.126:33045 recv error: Network error: failed to read from TLS socket (remote: 127.8.158.126:33045): Connection reset by peer (error 104)
2026-09-29T19:07:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1304 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708573736807-8825-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:07:06 dist-test-slave-nmsz krb5kdc[30439](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:07:06 dist-test-slave-nmsz krb5kdc[30439](info): set up 2 sockets
Sep 29 19:07:06 dist-test-slave-nmsz krb5kdc[30439](info): commencing operation
krb5kdc: starting...
W20260929 19:07:07.598420 8825 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Sep 29 19:07:07 dist-test-slave-nmsz krb5kdc[30439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:07:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:07:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.158.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
I20260929 19:07:07.710151 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:35557
--webserver_interface=127.8.158.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:36911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:35557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:07:07.787346 30452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:07:07.787593 30452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:07:07.787629 30452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:07:07.788758 30452 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:07:07.788836 30452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:07:07.788861 30452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:07:07.788882 30452 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:07:07.788911 30452 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:07:07.790350 30452 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:36911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:35557
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:35557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.158.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:07:07.790611 30452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:07:07.790823 30452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:07:07.795421 30460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:07:07.795459 30457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:07:07.795511 30458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:07:07.795706 30452 server_base.cc:1034] running on GCE node
I20260929 19:07:07.795925 30452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:07:07.796190 30452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:07:07.797325 30452 hybrid_clock.cc:648] HybridClock initialized: now 1790708827797305 us; error 38 us; skew 500 ppm
Sep 29 19:07:07 dist-test-slave-nmsz krb5kdc[30439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708827, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:07:07.804041 30452 init.cc:377] Logged in from keytab as kudu/127.8.158.126@KRBTEST.COM (short username kudu)
I20260929 19:07:07.804487 30452 webserver.cc:460] Webserver started at http://127.8.158.126:42913/ using document root <none> and password file <none>
I20260929 19:07:07.804695 30452 fs_manager.cc:359] Metadata directory not provided
I20260929 19:07:07.804739 30452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:07:07.804919 30452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:07:07.805929 30452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "debc57a21f3042a0bdbed6523d9b756e"
format_stamp: "Formatted at 2026-09-29 19:07:07 on dist-test-slave-nmsz"
server_key: "c9788d095c87be42b614835ab3aa4b8c"
server_key_iv: "820553513ece58fc10035fbc6cd925c1"
server_key_version: "encryptionkey@0"
I20260929 19:07:07.806273 30452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "debc57a21f3042a0bdbed6523d9b756e"
format_stamp: "Formatted at 2026-09-29 19:07:07 on dist-test-slave-nmsz"
server_key: "c9788d095c87be42b614835ab3aa4b8c"
server_key_iv: "820553513ece58fc10035fbc6cd925c1"
server_key_version: "encryptionkey@0"
I20260929 19:07:07.807600 30452 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:07:07.808393 30467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:07:07.808588 30452 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:07:07.808658 30452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "debc57a21f3042a0bdbed6523d9b756e"
format_stamp: "Formatted at 2026-09-29 19:07:07 on dist-test-slave-nmsz"
server_key: "c9788d095c87be42b614835ab3aa4b8c"
server_key_iv: "820553513ece58fc10035fbc6cd925c1"
server_key_version: "encryptionkey@0"
I20260929 19:07:07.808748 30452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:07:07.820330 30452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:07:07.822561 30452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:07:07.822714 30452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:07:07.826598 30452 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.126:35557
I20260929 19:07:07.826638 30519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.126:35557 every 8 connection(s)
I20260929 19:07:07.826949 30452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:07:07.827699 30520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:07:07.829772 30520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P debc57a21f3042a0bdbed6523d9b756e: Bootstrap starting.
I20260929 19:07:07.830348 30520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P debc57a21f3042a0bdbed6523d9b756e: Neither blocks nor log segments found. Creating new log.
I20260929 19:07:07.830621 30520 log.cc:824] T 00000000000000000000000000000000 P debc57a21f3042a0bdbed6523d9b756e: Log is configured to *not* fsync() on all Append() calls
I20260929 19:07:07.831264 30520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P debc57a21f3042a0bdbed6523d9b756e: No bootstrap required, opened a new log
I20260929 19:07:07.832825 30520 raft_consensus.cc:348] T 00000000000000000000000000000000 P debc57a21f3042a0bdbed6523d9b756e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "debc57a21f3042a0bdbed6523d9b756e" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35557 } }
I20260929 19:07:07.832971 30520 raft_consensus.cc:374] T 00000000000000000000000000000000 P debc57a21f3042a0bdbed6523d9b756e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:07:07.833019 30520 raft_consensus.cc:729] T 00000000000000000000000000000000 P debc57a21f3042a0bdbed6523d9b756e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: debc57a21f3042a0bdbed6523d9b756e, State: Initialized, Role: FOLLOWER
I20260929 19:07:07.833154 30520 consensus_queue.cc:260] T 00000000000000000000000000000000 P debc57a21f3042a0bdbed6523d9b756e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "debc57a21f3042a0bdbed6523d9b756e" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35557 } }
I20260929 19:07:07.833231 30520 raft_consensus.cc:388] T 00000000000000000000000000000000 P debc57a21f3042a0bdbed6523d9b756e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:07:07.833273 30520 raft_consensus.cc:482] T 00000000000000000000000000000000 P debc57a21f3042a0bdbed6523d9b756e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:07:07.833326 30520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P debc57a21f3042a0bdbed6523d9b756e [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:07:07.834091 30520 raft_consensus.cc:504] T 00000000000000000000000000000000 P debc57a21f3042a0bdbed6523d9b756e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "debc57a21f3042a0bdbed6523d9b756e" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35557 } }
I20260929 19:07:07.834218 30520 leader_election.cc:302] T 00000000000000000000000000000000 P debc57a21f3042a0bdbed6523d9b756e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: debc57a21f3042a0bdbed6523d9b756e; no voters:
I20260929 19:07:07.834388 30520 leader_election.cc:288] T 00000000000000000000000000000000 P debc57a21f3042a0bdbed6523d9b756e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:07:07.834370 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 30452
I20260929 19:07:07.834421 30523 raft_consensus.cc:2781] T 00000000000000000000000000000000 P debc57a21f3042a0bdbed6523d9b756e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:07:07.834478 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:07:07.834568 30523 raft_consensus.cc:686] T 00000000000000000000000000000000 P debc57a21f3042a0bdbed6523d9b756e [term 1 LEADER]: Becoming Leader. State: Replica: debc57a21f3042a0bdbed6523d9b756e, State: Running, Role: LEADER
I20260929 19:07:07.834625 30520 sys_catalog.cc:564] T 00000000000000000000000000000000 P debc57a21f3042a0bdbed6523d9b756e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:07:07.834659 8825 external_mini_cluster.cc:1428] Setting key e352a72376ad94689c3ea970998061a6
I20260929 19:07:07.834633 30523 consensus_queue.cc:237] T 00000000000000000000000000000000 P debc57a21f3042a0bdbed6523d9b756e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "debc57a21f3042a0bdbed6523d9b756e" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35557 } }
I20260929 19:07:07.835127 30525 sys_catalog.cc:455] T 00000000000000000000000000000000 P debc57a21f3042a0bdbed6523d9b756e [sys.catalog]: SysCatalogTable state changed. Reason: New leader debc57a21f3042a0bdbed6523d9b756e. Latest consensus state: current_term: 1 leader_uuid: "debc57a21f3042a0bdbed6523d9b756e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "debc57a21f3042a0bdbed6523d9b756e" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35557 } } }
I20260929 19:07:07.835206 30525 sys_catalog.cc:458] T 00000000000000000000000000000000 P debc57a21f3042a0bdbed6523d9b756e [sys.catalog]: This master's current role is: LEADER
I20260929 19:07:07.835369 30524 sys_catalog.cc:455] T 00000000000000000000000000000000 P debc57a21f3042a0bdbed6523d9b756e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "debc57a21f3042a0bdbed6523d9b756e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "debc57a21f3042a0bdbed6523d9b756e" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35557 } } }
I20260929 19:07:07.835448 30524 sys_catalog.cc:458] T 00000000000000000000000000000000 P debc57a21f3042a0bdbed6523d9b756e [sys.catalog]: This master's current role is: LEADER
I20260929 19:07:07.835573 30532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:07:07.836081 30532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:07:07.837550 30532 catalog_manager.cc:1347] Generated new cluster ID: ed3fbb1c9fc34a918e7a0930a9ab532e
I20260929 19:07:07.837602 30532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:07:07.840612 30534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:07.835242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38845 (local address 127.8.158.126:35557)
0929 19:07:07.836115 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:07.836123 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:07.836158 (+ 35us) server_negotiation.cc:407] Connection header received
0929 19:07:07.836272 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:07.836280 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:07.836337 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:07.836514 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:07.836727 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:07.837947 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:07.838791 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:07.839207 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:07.839341 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:07.840004 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:07.840008 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:07.840008 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:07:07.840176 (+ 168us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:07.840203 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:07:07.840213 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:07.840312 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:07.840388 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 19:07:07.840415 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":83,"threads_started":1}
I20260929 19:07:07.841404 8825 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:07:07.845496 30534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:07.842672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40659 (local address 127.8.158.126:35557)
0929 19:07:07.842708 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:07.842710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:07.842819 (+ 109us) server_negotiation.cc:407] Connection header received
0929 19:07:07.842888 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:07.842889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:07.842921 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:07.842976 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:07.843097 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:07.843611 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:07.844316 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:07.844540 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:07.844612 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:07.845253 (+ 641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:07.845254 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:07.845255 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:07:07.845259 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:07.845271 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:07:07.845274 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:07.845336 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:07.845396 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 19:07:07.845408 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 19:07:07.852484 30532 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:07:07.852980 30532 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:07:07.861354 30532 catalog_manager.cc:5868] T 00000000000000000000000000000000 P debc57a21f3042a0bdbed6523d9b756e: Generated new TSK 0
I20260929 19:07:07.861562 30532 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:07:07.864274 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 30452
W20260929 19:07:07.868305 30437 connection.cc:535] client connection to 127.8.158.126:35557 recv error: Network error: failed to read from TLS socket (remote: 127.8.158.126:35557): Connection reset by peer (error 104)
2026-09-29T19:07:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1304 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708573736807-8825-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:07:07 dist-test-slave-nmsz krb5kdc[30551](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:07:07 dist-test-slave-nmsz krb5kdc[30551](info): set up 2 sockets
Sep 29 19:07:07 dist-test-slave-nmsz krb5kdc[30551](info): commencing operation
krb5kdc: starting...
W20260929 19:07:08.890271 8825 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Sep 29 19:07:08 dist-test-slave-nmsz krb5kdc[30551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:07:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:07:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.158.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
I20260929 19:07:09.008675 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:42171
--webserver_interface=127.8.158.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:46357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:42171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:07:09.086748 30564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:07:09.086937 30564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:07:09.086961 30564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:07:09.088085 30564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:07:09.088126 30564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:07:09.088142 30564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:07:09.088155 30564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:07:09.088205 30564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:07:09.089610 30564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:46357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:42171
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:42171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.158.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:07:09.089850 30564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:07:09.090066 30564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:07:09.094635 30570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:07:09.094692 30569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:07:09.094660 30572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:07:09.095000 30564 server_base.cc:1034] running on GCE node
I20260929 19:07:09.095217 30564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:07:09.095516 30564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:07:09.096650 30564 hybrid_clock.cc:648] HybridClock initialized: now 1790708829096632 us; error 37 us; skew 500 ppm
Sep 29 19:07:09 dist-test-slave-nmsz krb5kdc[30551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708829, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:07:09.099037 30564 init.cc:377] Logged in from keytab as kudu/127.8.158.126@KRBTEST.COM (short username kudu)
I20260929 19:07:09.099480 30564 webserver.cc:460] Webserver started at http://127.8.158.126:35279/ using document root <none> and password file <none>
I20260929 19:07:09.099696 30564 fs_manager.cc:359] Metadata directory not provided
I20260929 19:07:09.099740 30564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:07:09.099900 30564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:07:09.100852 30564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "c02ce98e5fb74e978f20b89d18492dc7"
format_stamp: "Formatted at 2026-09-29 19:07:09 on dist-test-slave-nmsz"
server_key: "33a7e068be004662f7752eeb34ae0db4"
server_key_iv: "25e3e824261484e54bdf5300f5cdc54f"
server_key_version: "encryptionkey@0"
I20260929 19:07:09.101193 30564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "c02ce98e5fb74e978f20b89d18492dc7"
format_stamp: "Formatted at 2026-09-29 19:07:09 on dist-test-slave-nmsz"
server_key: "33a7e068be004662f7752eeb34ae0db4"
server_key_iv: "25e3e824261484e54bdf5300f5cdc54f"
server_key_version: "encryptionkey@0"
I20260929 19:07:09.102473 30564 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 19:07:09.103223 30579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:07:09.103417 30564 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:07:09.103487 30564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "c02ce98e5fb74e978f20b89d18492dc7"
format_stamp: "Formatted at 2026-09-29 19:07:09 on dist-test-slave-nmsz"
server_key: "33a7e068be004662f7752eeb34ae0db4"
server_key_iv: "25e3e824261484e54bdf5300f5cdc54f"
server_key_version: "encryptionkey@0"
I20260929 19:07:09.103574 30564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:07:09.106771 30564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:07:09.108566 30564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:07:09.108668 30564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:07:09.112671 30564 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.126:42171
I20260929 19:07:09.112759 30631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.126:42171 every 8 connection(s)
I20260929 19:07:09.113056 30564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:07:09.113961 30632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:07:09.116070 30632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c02ce98e5fb74e978f20b89d18492dc7: Bootstrap starting.
I20260929 19:07:09.116566 30632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c02ce98e5fb74e978f20b89d18492dc7: Neither blocks nor log segments found. Creating new log.
I20260929 19:07:09.116788 30632 log.cc:824] T 00000000000000000000000000000000 P c02ce98e5fb74e978f20b89d18492dc7: Log is configured to *not* fsync() on all Append() calls
I20260929 19:07:09.117405 30632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c02ce98e5fb74e978f20b89d18492dc7: No bootstrap required, opened a new log
I20260929 19:07:09.118986 30632 raft_consensus.cc:348] T 00000000000000000000000000000000 P c02ce98e5fb74e978f20b89d18492dc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c02ce98e5fb74e978f20b89d18492dc7" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 42171 } }
I20260929 19:07:09.119114 30632 raft_consensus.cc:374] T 00000000000000000000000000000000 P c02ce98e5fb74e978f20b89d18492dc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:07:09.119136 30632 raft_consensus.cc:729] T 00000000000000000000000000000000 P c02ce98e5fb74e978f20b89d18492dc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c02ce98e5fb74e978f20b89d18492dc7, State: Initialized, Role: FOLLOWER
I20260929 19:07:09.119263 30632 consensus_queue.cc:260] T 00000000000000000000000000000000 P c02ce98e5fb74e978f20b89d18492dc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c02ce98e5fb74e978f20b89d18492dc7" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 42171 } }
I20260929 19:07:09.119334 30632 raft_consensus.cc:388] T 00000000000000000000000000000000 P c02ce98e5fb74e978f20b89d18492dc7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:07:09.119355 30632 raft_consensus.cc:482] T 00000000000000000000000000000000 P c02ce98e5fb74e978f20b89d18492dc7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:07:09.119382 30632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c02ce98e5fb74e978f20b89d18492dc7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:07:09.120127 30632 raft_consensus.cc:504] T 00000000000000000000000000000000 P c02ce98e5fb74e978f20b89d18492dc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c02ce98e5fb74e978f20b89d18492dc7" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 42171 } }
I20260929 19:07:09.120208 30632 leader_election.cc:302] T 00000000000000000000000000000000 P c02ce98e5fb74e978f20b89d18492dc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c02ce98e5fb74e978f20b89d18492dc7; no voters:
I20260929 19:07:09.120321 30632 leader_election.cc:288] T 00000000000000000000000000000000 P c02ce98e5fb74e978f20b89d18492dc7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:07:09.120389 30635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c02ce98e5fb74e978f20b89d18492dc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:07:09.120573 30635 raft_consensus.cc:686] T 00000000000000000000000000000000 P c02ce98e5fb74e978f20b89d18492dc7 [term 1 LEADER]: Becoming Leader. State: Replica: c02ce98e5fb74e978f20b89d18492dc7, State: Running, Role: LEADER
I20260929 19:07:09.120601 30632 sys_catalog.cc:564] T 00000000000000000000000000000000 P c02ce98e5fb74e978f20b89d18492dc7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:07:09.120692 30635 consensus_queue.cc:237] T 00000000000000000000000000000000 P c02ce98e5fb74e978f20b89d18492dc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c02ce98e5fb74e978f20b89d18492dc7" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 42171 } }
I20260929 19:07:09.121089 30636 sys_catalog.cc:455] T 00000000000000000000000000000000 P c02ce98e5fb74e978f20b89d18492dc7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c02ce98e5fb74e978f20b89d18492dc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c02ce98e5fb74e978f20b89d18492dc7" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 42171 } } }
I20260929 19:07:09.121122 30637 sys_catalog.cc:455] T 00000000000000000000000000000000 P c02ce98e5fb74e978f20b89d18492dc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c02ce98e5fb74e978f20b89d18492dc7. Latest consensus state: current_term: 1 leader_uuid: "c02ce98e5fb74e978f20b89d18492dc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c02ce98e5fb74e978f20b89d18492dc7" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 42171 } } }
I20260929 19:07:09.121253 30636 sys_catalog.cc:458] T 00000000000000000000000000000000 P c02ce98e5fb74e978f20b89d18492dc7 [sys.catalog]: This master's current role is: LEADER
I20260929 19:07:09.121655 30637 sys_catalog.cc:458] T 00000000000000000000000000000000 P c02ce98e5fb74e978f20b89d18492dc7 [sys.catalog]: This master's current role is: LEADER
I20260929 19:07:09.121762 30644 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:07:09.122212 30644 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:07:09.122956 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 30564
I20260929 19:07:09.123065 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:07:09.123260 8825 external_mini_cluster.cc:1428] Setting key 198dca42942a6c48dd5f04c11e84279e
I20260929 19:07:09.123728 30644 catalog_manager.cc:1347] Generated new cluster ID: 4a390d7df97c49c5933177e59b5c571f
I20260929 19:07:09.123775 30644 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:07:09.128121 30655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:09.123694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55147 (local address 127.8.158.126:42171)
0929 19:07:09.123845 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:09.123854 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:09.126681 (+ 2827us) server_negotiation.cc:407] Connection header received
0929 19:07:09.126802 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:09.126809 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:09.126838 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:09.126973 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:09.127689 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:09.127692 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:09.127693 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:07:09.127698 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:09.127744 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:07:09.127758 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:09.127830 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:09.127889 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 19:07:09.127936 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":66,"threads_started":1}
I20260929 19:07:09.129176 8825 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:07:09.136526 30655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:09.135017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35943 (local address 127.8.158.126:42171)
0929 19:07:09.135051 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:09.135052 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:09.135416 (+ 364us) server_negotiation.cc:407] Connection header received
0929 19:07:09.135510 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:09.135511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:09.135516 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:09.135544 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:09.136275 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:09.136276 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:09.136276 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:07:09.136280 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:09.136305 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:07:09.136309 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:09.136339 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:09.136443 (+ 104us) server_negotiation.cc:299] Negotiation successful
0929 19:07:09.136456 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 19:07:09.158815 30644 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:07:09.158989 30644 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:07:09.170658 30644 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c02ce98e5fb74e978f20b89d18492dc7: Generated new TSK 0
I20260929 19:07:09.170849 30644 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:07:09.176034 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 30564
2026-09-29T19:07:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1311 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708573736807-8825-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:07:09 dist-test-slave-nmsz krb5kdc[30663](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:07:09 dist-test-slave-nmsz krb5kdc[30663](info): set up 2 sockets
Sep 29 19:07:09 dist-test-slave-nmsz krb5kdc[30663](info): commencing operation
krb5kdc: starting...
W20260929 19:07:10.206375 8825 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Sep 29 19:07:10 dist-test-slave-nmsz krb5kdc[30663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:07:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:07:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.158.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
I20260929 19:07:10.317112 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:43119
--webserver_interface=127.8.158.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:41515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:43119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:07:10.394428 30676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:07:10.394656 30676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:07:10.394697 30676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:07:10.395819 30676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:07:10.395890 30676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:07:10.395916 30676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:07:10.395948 30676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:07:10.395972 30676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:07:10.397400 30676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:41515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:43119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:43119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.158.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:07:10.397658 30676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:07:10.397881 30676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:07:10.402557 30684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:07:10.402565 30681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:07:10.402585 30676 server_base.cc:1034] running on GCE node
W20260929 19:07:10.402566 30682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:07:10.402985 30676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:07:10.403209 30676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:07:10.404374 30676 hybrid_clock.cc:648] HybridClock initialized: now 1790708830404364 us; error 39 us; skew 500 ppm
Sep 29 19:07:10 dist-test-slave-nmsz krb5kdc[30663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708830, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:07:10.406934 30676 init.cc:377] Logged in from keytab as kudu/127.8.158.126@KRBTEST.COM (short username kudu)
I20260929 19:07:10.407410 30676 webserver.cc:460] Webserver started at http://127.8.158.126:46111/ using document root <none> and password file <none>
I20260929 19:07:10.407631 30676 fs_manager.cc:359] Metadata directory not provided
I20260929 19:07:10.407698 30676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:07:10.407843 30676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:07:10.408812 30676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "22e919499d6845299c97720416b443ec"
format_stamp: "Formatted at 2026-09-29 19:07:10 on dist-test-slave-nmsz"
server_key: "aac67f22ecc35064e452a7d57ce716cc"
server_key_iv: "ca50cb5f343c6eff22c74003517e45fb"
server_key_version: "encryptionkey@0"
I20260929 19:07:10.409164 30676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "22e919499d6845299c97720416b443ec"
format_stamp: "Formatted at 2026-09-29 19:07:10 on dist-test-slave-nmsz"
server_key: "aac67f22ecc35064e452a7d57ce716cc"
server_key_iv: "ca50cb5f343c6eff22c74003517e45fb"
server_key_version: "encryptionkey@0"
I20260929 19:07:10.410476 30676 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:07:10.411202 30691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:07:10.411374 30676 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:07:10.411470 30676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "22e919499d6845299c97720416b443ec"
format_stamp: "Formatted at 2026-09-29 19:07:10 on dist-test-slave-nmsz"
server_key: "aac67f22ecc35064e452a7d57ce716cc"
server_key_iv: "ca50cb5f343c6eff22c74003517e45fb"
server_key_version: "encryptionkey@0"
I20260929 19:07:10.411554 30676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:07:10.428404 30676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:07:10.430178 30676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:07:10.430387 30676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:07:10.434134 30676 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.126:43119
I20260929 19:07:10.434186 30743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.126:43119 every 8 connection(s)
I20260929 19:07:10.434526 30676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:07:10.435282 30744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:07:10.437295 30744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22e919499d6845299c97720416b443ec: Bootstrap starting.
I20260929 19:07:10.437788 30744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22e919499d6845299c97720416b443ec: Neither blocks nor log segments found. Creating new log.
I20260929 19:07:10.438012 30744 log.cc:824] T 00000000000000000000000000000000 P 22e919499d6845299c97720416b443ec: Log is configured to *not* fsync() on all Append() calls
I20260929 19:07:10.438630 30744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22e919499d6845299c97720416b443ec: No bootstrap required, opened a new log
I20260929 19:07:10.440063 30744 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22e919499d6845299c97720416b443ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22e919499d6845299c97720416b443ec" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 43119 } }
I20260929 19:07:10.440187 30744 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22e919499d6845299c97720416b443ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:07:10.440208 30744 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22e919499d6845299c97720416b443ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22e919499d6845299c97720416b443ec, State: Initialized, Role: FOLLOWER
I20260929 19:07:10.440299 30744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22e919499d6845299c97720416b443ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22e919499d6845299c97720416b443ec" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 43119 } }
I20260929 19:07:10.440347 30744 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22e919499d6845299c97720416b443ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:07:10.440367 30744 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22e919499d6845299c97720416b443ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:07:10.440441 30744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22e919499d6845299c97720416b443ec [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:07:10.440950 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 30676
I20260929 19:07:10.441042 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:07:10.441213 8825 external_mini_cluster.cc:1428] Setting key 80ec5508c6e97a4ece788dff56cd3ce6
I20260929 19:07:10.441810 30744 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22e919499d6845299c97720416b443ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22e919499d6845299c97720416b443ec" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 43119 } }
I20260929 19:07:10.441980 30744 leader_election.cc:302] T 00000000000000000000000000000000 P 22e919499d6845299c97720416b443ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22e919499d6845299c97720416b443ec; no voters:
I20260929 19:07:10.442163 30744 leader_election.cc:288] T 00000000000000000000000000000000 P 22e919499d6845299c97720416b443ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:07:10.442188 30749 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22e919499d6845299c97720416b443ec [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:07:10.442293 30749 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22e919499d6845299c97720416b443ec [term 1 LEADER]: Becoming Leader. State: Replica: 22e919499d6845299c97720416b443ec, State: Running, Role: LEADER
I20260929 19:07:10.442433 30744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22e919499d6845299c97720416b443ec [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:07:10.442440 30749 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22e919499d6845299c97720416b443ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22e919499d6845299c97720416b443ec" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 43119 } }
I20260929 19:07:10.442804 30749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22e919499d6845299c97720416b443ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22e919499d6845299c97720416b443ec. Latest consensus state: current_term: 1 leader_uuid: "22e919499d6845299c97720416b443ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22e919499d6845299c97720416b443ec" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 43119 } } }
I20260929 19:07:10.442821 30751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22e919499d6845299c97720416b443ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22e919499d6845299c97720416b443ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22e919499d6845299c97720416b443ec" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 43119 } } }
I20260929 19:07:10.443032 30751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22e919499d6845299c97720416b443ec [sys.catalog]: This master's current role is: LEADER
I20260929 19:07:10.442961 30749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22e919499d6845299c97720416b443ec [sys.catalog]: This master's current role is: LEADER
I20260929 19:07:10.443487 30758 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:07:10.443979 30758 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:07:10.445715 30758 catalog_manager.cc:1347] Generated new cluster ID: a6cdf57e38954f1f95588537fdbbbf2e
I20260929 19:07:10.445770 30758 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 19:07:10 dist-test-slave-nmsz krb5kdc[30663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:07:10.451640 30748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:10.441722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59985 (local address 127.8.158.126:43119)
0929 19:07:10.441963 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:10.441970 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:10.441994 (+ 24us) server_negotiation.cc:407] Connection header received
0929 19:07:10.442107 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:10.442114 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:10.442164 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:10.443155 (+ 991us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:10.443268 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:10.444624 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:10.445667 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:10.446120 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:10.446238 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:10.448657 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:10.448664 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:10.448664 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:07:10.448682 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:10.450167 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:10.450567 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:10.450570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:10.450571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:10.450656 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:10.450911 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:10.450912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:10.450913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:10.451125 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:10.451264 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:10.451369 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 19:07:10.451498 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":155,"threads_started":1}
I20260929 19:07:10.452486 8825 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:07:10.460534 30748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:10.453771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35001 (local address 127.8.158.126:43119)
0929 19:07:10.453817 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:10.453818 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:10.453882 (+ 64us) server_negotiation.cc:407] Connection header received
0929 19:07:10.453937 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:10.453938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:10.453946 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:10.453988 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:10.454560 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:10.455226 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:10.456165 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:10.456487 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:10.456564 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:10.457800 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:10.457802 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:10.457802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:07:10.457808 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:10.459431 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:10.459789 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:10.459790 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:10.459791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:10.459850 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:10.460052 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:10.460053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:10.460053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:10.460175 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:10.460272 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:10.460371 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 19:07:10.460450 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 19:07:10.473786 30758 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:07:10.474265 30758 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:07:10.486959 30758 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22e919499d6845299c97720416b443ec: Generated new TSK 0
I20260929 19:07:10.487134 30758 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:07:10.489919 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 30676
2026-09-29T19:07:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1313 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708573736807-8825-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:07:10 dist-test-slave-nmsz krb5kdc[30775](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:07:10 dist-test-slave-nmsz krb5kdc[30775](info): set up 2 sockets
Sep 29 19:07:10 dist-test-slave-nmsz krb5kdc[30775](info): commencing operation
krb5kdc: starting...
W20260929 19:07:11.522711 8825 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Sep 29 19:07:11 dist-test-slave-nmsz krb5kdc[30775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:07:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:07:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.158.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
I20260929 19:07:11.637706 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:44261
--webserver_interface=127.8.158.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:37197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:44261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:07:11.719751 30788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:07:11.719964 30788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:07:11.719988 30788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:07:11.721156 30788 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:07:11.721191 30788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:07:11.721207 30788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:07:11.721221 30788 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:07:11.721235 30788 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:07:11.722864 30788 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:37197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:44261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:44261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.158.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:07:11.723050 30788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:07:11.723219 30788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:07:11.728173 30796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:07:11.728235 30788 server_base.cc:1034] running on GCE node
W20260929 19:07:11.728179 30793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:07:11.728180 30794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:07:11.728648 30788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:07:11.728901 30788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:07:11.730044 30788 hybrid_clock.cc:648] HybridClock initialized: now 1790708831730024 us; error 41 us; skew 500 ppm
Sep 29 19:07:11 dist-test-slave-nmsz krb5kdc[30775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708831, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:07:11.732610 30788 init.cc:377] Logged in from keytab as kudu/127.8.158.126@KRBTEST.COM (short username kudu)
I20260929 19:07:11.733088 30788 webserver.cc:460] Webserver started at http://127.8.158.126:40673/ using document root <none> and password file <none>
I20260929 19:07:11.733320 30788 fs_manager.cc:359] Metadata directory not provided
I20260929 19:07:11.733392 30788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:07:11.733542 30788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:07:11.734700 30788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "030af8acef7b4c7c955485549459fb61"
format_stamp: "Formatted at 2026-09-29 19:07:11 on dist-test-slave-nmsz"
server_key: "872a664ce1d564bb2728fe6d1a100f21"
server_key_iv: "d04f58e23ca63d9e20633dcedaa22eba"
server_key_version: "encryptionkey@0"
I20260929 19:07:11.735064 30788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "030af8acef7b4c7c955485549459fb61"
format_stamp: "Formatted at 2026-09-29 19:07:11 on dist-test-slave-nmsz"
server_key: "872a664ce1d564bb2728fe6d1a100f21"
server_key_iv: "d04f58e23ca63d9e20633dcedaa22eba"
server_key_version: "encryptionkey@0"
I20260929 19:07:11.736402 30788 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:07:11.737103 30803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:07:11.737282 30788 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:07:11.737382 30788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "030af8acef7b4c7c955485549459fb61"
format_stamp: "Formatted at 2026-09-29 19:07:11 on dist-test-slave-nmsz"
server_key: "872a664ce1d564bb2728fe6d1a100f21"
server_key_iv: "d04f58e23ca63d9e20633dcedaa22eba"
server_key_version: "encryptionkey@0"
I20260929 19:07:11.737465 30788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:07:11.746860 30788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:07:11.748620 30788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:07:11.748787 30788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:07:11.753000 30788 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.126:44261
I20260929 19:07:11.753028 30855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.126:44261 every 8 connection(s)
I20260929 19:07:11.753371 30788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:07:11.754144 30856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:07:11.756217 30856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 030af8acef7b4c7c955485549459fb61: Bootstrap starting.
I20260929 19:07:11.756692 30856 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 030af8acef7b4c7c955485549459fb61: Neither blocks nor log segments found. Creating new log.
I20260929 19:07:11.756922 30856 log.cc:824] T 00000000000000000000000000000000 P 030af8acef7b4c7c955485549459fb61: Log is configured to *not* fsync() on all Append() calls
I20260929 19:07:11.757540 30856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 030af8acef7b4c7c955485549459fb61: No bootstrap required, opened a new log
I20260929 19:07:11.759097 30856 raft_consensus.cc:348] T 00000000000000000000000000000000 P 030af8acef7b4c7c955485549459fb61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "030af8acef7b4c7c955485549459fb61" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 44261 } }
I20260929 19:07:11.759222 30856 raft_consensus.cc:374] T 00000000000000000000000000000000 P 030af8acef7b4c7c955485549459fb61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:07:11.759243 30856 raft_consensus.cc:729] T 00000000000000000000000000000000 P 030af8acef7b4c7c955485549459fb61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 030af8acef7b4c7c955485549459fb61, State: Initialized, Role: FOLLOWER
I20260929 19:07:11.759367 30856 consensus_queue.cc:260] T 00000000000000000000000000000000 P 030af8acef7b4c7c955485549459fb61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "030af8acef7b4c7c955485549459fb61" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 44261 } }
I20260929 19:07:11.759440 30856 raft_consensus.cc:388] T 00000000000000000000000000000000 P 030af8acef7b4c7c955485549459fb61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:07:11.759462 30856 raft_consensus.cc:482] T 00000000000000000000000000000000 P 030af8acef7b4c7c955485549459fb61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:07:11.759493 30856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 030af8acef7b4c7c955485549459fb61 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:07:11.760221 30856 raft_consensus.cc:504] T 00000000000000000000000000000000 P 030af8acef7b4c7c955485549459fb61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "030af8acef7b4c7c955485549459fb61" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 44261 } }
I20260929 19:07:11.760298 30856 leader_election.cc:302] T 00000000000000000000000000000000 P 030af8acef7b4c7c955485549459fb61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 030af8acef7b4c7c955485549459fb61; no voters:
I20260929 19:07:11.760404 30856 leader_election.cc:288] T 00000000000000000000000000000000 P 030af8acef7b4c7c955485549459fb61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:07:11.760484 30859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 030af8acef7b4c7c955485549459fb61 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:07:11.760659 30859 raft_consensus.cc:686] T 00000000000000000000000000000000 P 030af8acef7b4c7c955485549459fb61 [term 1 LEADER]: Becoming Leader. State: Replica: 030af8acef7b4c7c955485549459fb61, State: Running, Role: LEADER
I20260929 19:07:11.760663 30856 sys_catalog.cc:564] T 00000000000000000000000000000000 P 030af8acef7b4c7c955485549459fb61 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:07:11.760763 30859 consensus_queue.cc:237] T 00000000000000000000000000000000 P 030af8acef7b4c7c955485549459fb61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "030af8acef7b4c7c955485549459fb61" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 44261 } }
I20260929 19:07:11.761193 30861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 030af8acef7b4c7c955485549459fb61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 030af8acef7b4c7c955485549459fb61. Latest consensus state: current_term: 1 leader_uuid: "030af8acef7b4c7c955485549459fb61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "030af8acef7b4c7c955485549459fb61" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 44261 } } }
I20260929 19:07:11.761287 30861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 030af8acef7b4c7c955485549459fb61 [sys.catalog]: This master's current role is: LEADER
I20260929 19:07:11.761448 30860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 030af8acef7b4c7c955485549459fb61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "030af8acef7b4c7c955485549459fb61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "030af8acef7b4c7c955485549459fb61" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 44261 } } }
I20260929 19:07:11.761502 30868 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:07:11.762179 30868 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:07:11.762238 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 30788
I20260929 19:07:11.762368 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:07:11.762575 8825 external_mini_cluster.cc:1428] Setting key ad004c66cbff4e910d02d447303a250b
I20260929 19:07:11.761538 30860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 030af8acef7b4c7c955485549459fb61 [sys.catalog]: This master's current role is: LEADER
I20260929 19:07:11.763818 30868 catalog_manager.cc:1347] Generated new cluster ID: cc5e32b4779341ad89efb5c99cb79127
I20260929 19:07:11.763878 30868 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:07:11.768836 30878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:11.763044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52931 (local address 127.8.158.126:44261)
0929 19:07:11.763195 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:11.763205 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:11.763299 (+ 94us) server_negotiation.cc:407] Connection header received
0929 19:07:11.763466 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:11.763474 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:11.763536 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:11.763680 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:11.763965 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:11.765477 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:11.766615 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:11.767244 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:11.767353 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:11.768102 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:11.768106 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:11.768106 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:07:11.768327 (+ 221us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:11.768364 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:07:11.768379 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:11.768442 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:11.768566 (+ 124us) server_negotiation.cc:299] Negotiation successful
0929 19:07:11.768630 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":73,"threads_started":1}
I20260929 19:07:11.769713 8825 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:07:11.774600 30878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:11.771291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57289 (local address 127.8.158.126:44261)
0929 19:07:11.771322 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:11.771323 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:11.771332 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:07:11.771375 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:11.771376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:11.771384 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:11.771437 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:11.771558 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:11.772330 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:11.773319 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:11.773635 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:11.773697 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:11.774369 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:11.774370 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:11.774371 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:07:11.774376 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:11.774389 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:07:11.774393 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:11.774448 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:11.774508 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 19:07:11.774530 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260929 19:07:11.786782 30868 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:07:11.787532 30868 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:07:11.792608 30868 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 030af8acef7b4c7c955485549459fb61: Generated new TSK 0
I20260929 19:07:11.792776 30868 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:07:11.796573 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 30788
2026-09-29T19:07:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1306 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708573736807-8825-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:07:11 dist-test-slave-nmsz krb5kdc[30887](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:07:11 dist-test-slave-nmsz krb5kdc[30887](info): set up 2 sockets
Sep 29 19:07:11 dist-test-slave-nmsz krb5kdc[30887](info): commencing operation
krb5kdc: starting...
W20260929 19:07:12.826465 8825 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Sep 29 19:07:12 dist-test-slave-nmsz krb5kdc[30887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:07:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:07:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.158.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
I20260929 19:07:12.937397 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:39969
--webserver_interface=127.8.158.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:39969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:07:13.014948 30900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:07:13.015169 30900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:07:13.015208 30900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:07:13.016330 30900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:07:13.016394 30900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:07:13.016420 30900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:07:13.016448 30900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:07:13.016479 30900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:07:13.017875 30900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:39969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:39969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.158.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:07:13.018132 30900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:07:13.018369 30900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:07:13.022925 30905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:07:13.022936 30908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:07:13.023113 30900 server_base.cc:1034] running on GCE node
W20260929 19:07:13.022946 30906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:07:13.023422 30900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:07:13.023661 30900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:07:13.024803 30900 hybrid_clock.cc:648] HybridClock initialized: now 1790708833024787 us; error 30 us; skew 500 ppm
Sep 29 19:07:13 dist-test-slave-nmsz krb5kdc[30887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708833, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:07:13.030848 30900 init.cc:377] Logged in from keytab as kudu/127.8.158.126@KRBTEST.COM (short username kudu)
I20260929 19:07:13.031275 30900 webserver.cc:460] Webserver started at http://127.8.158.126:44571/ using document root <none> and password file <none>
I20260929 19:07:13.031445 30900 fs_manager.cc:359] Metadata directory not provided
I20260929 19:07:13.031482 30900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:07:13.031565 30900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:07:13.032500 30900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "85c815a50e0e4b3688b6a8931383ebb9"
format_stamp: "Formatted at 2026-09-29 19:07:13 on dist-test-slave-nmsz"
server_key: "efaa895d55c178068c72d626406cb57b"
server_key_iv: "2ca133d5e3b22d8a64377ce9b54bd1e2"
server_key_version: "encryptionkey@0"
I20260929 19:07:13.032787 30900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "85c815a50e0e4b3688b6a8931383ebb9"
format_stamp: "Formatted at 2026-09-29 19:07:13 on dist-test-slave-nmsz"
server_key: "efaa895d55c178068c72d626406cb57b"
server_key_iv: "2ca133d5e3b22d8a64377ce9b54bd1e2"
server_key_version: "encryptionkey@0"
I20260929 19:07:13.034020 30900 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 19:07:13.034795 30915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:07:13.035017 30900 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 19:07:13.035116 30900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "85c815a50e0e4b3688b6a8931383ebb9"
format_stamp: "Formatted at 2026-09-29 19:07:13 on dist-test-slave-nmsz"
server_key: "efaa895d55c178068c72d626406cb57b"
server_key_iv: "2ca133d5e3b22d8a64377ce9b54bd1e2"
server_key_version: "encryptionkey@0"
I20260929 19:07:13.035198 30900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:07:13.038353 30900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:07:13.040097 30900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:07:13.040251 30900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:07:13.044166 30900 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.126:39969
I20260929 19:07:13.044193 30967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.126:39969 every 8 connection(s)
I20260929 19:07:13.044509 30900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:07:13.045447 30968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:07:13.047640 30968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85c815a50e0e4b3688b6a8931383ebb9: Bootstrap starting.
I20260929 19:07:13.048175 30968 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85c815a50e0e4b3688b6a8931383ebb9: Neither blocks nor log segments found. Creating new log.
I20260929 19:07:13.048442 30968 log.cc:824] T 00000000000000000000000000000000 P 85c815a50e0e4b3688b6a8931383ebb9: Log is configured to *not* fsync() on all Append() calls
I20260929 19:07:13.049115 30968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85c815a50e0e4b3688b6a8931383ebb9: No bootstrap required, opened a new log
I20260929 19:07:13.050698 30968 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85c815a50e0e4b3688b6a8931383ebb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c815a50e0e4b3688b6a8931383ebb9" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 39969 } }
I20260929 19:07:13.050835 30968 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85c815a50e0e4b3688b6a8931383ebb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:07:13.050858 30968 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85c815a50e0e4b3688b6a8931383ebb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85c815a50e0e4b3688b6a8931383ebb9, State: Initialized, Role: FOLLOWER
I20260929 19:07:13.051003 30968 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85c815a50e0e4b3688b6a8931383ebb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c815a50e0e4b3688b6a8931383ebb9" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 39969 } }
I20260929 19:07:13.051084 30968 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85c815a50e0e4b3688b6a8931383ebb9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:07:13.051157 30968 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85c815a50e0e4b3688b6a8931383ebb9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:07:13.051213 30968 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85c815a50e0e4b3688b6a8931383ebb9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:07:13.051266 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 30900
I20260929 19:07:13.051367 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:07:13.051566 8825 external_mini_cluster.cc:1428] Setting key c580a3777feb522ca658fc0c6a469f51
I20260929 19:07:13.052301 30968 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85c815a50e0e4b3688b6a8931383ebb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c815a50e0e4b3688b6a8931383ebb9" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 39969 } }
I20260929 19:07:13.052418 30968 leader_election.cc:302] T 00000000000000000000000000000000 P 85c815a50e0e4b3688b6a8931383ebb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85c815a50e0e4b3688b6a8931383ebb9; no voters:
I20260929 19:07:13.052596 30968 leader_election.cc:288] T 00000000000000000000000000000000 P 85c815a50e0e4b3688b6a8931383ebb9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:07:13.052637 30973 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85c815a50e0e4b3688b6a8931383ebb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:07:13.052892 30968 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85c815a50e0e4b3688b6a8931383ebb9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:07:13.052934 30973 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85c815a50e0e4b3688b6a8931383ebb9 [term 1 LEADER]: Becoming Leader. State: Replica: 85c815a50e0e4b3688b6a8931383ebb9, State: Running, Role: LEADER
I20260929 19:07:13.053089 30973 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85c815a50e0e4b3688b6a8931383ebb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c815a50e0e4b3688b6a8931383ebb9" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 39969 } }
I20260929 19:07:13.053488 30974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85c815a50e0e4b3688b6a8931383ebb9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85c815a50e0e4b3688b6a8931383ebb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c815a50e0e4b3688b6a8931383ebb9" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 39969 } } }
I20260929 19:07:13.053581 30974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85c815a50e0e4b3688b6a8931383ebb9 [sys.catalog]: This master's current role is: LEADER
I20260929 19:07:13.053478 30975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85c815a50e0e4b3688b6a8931383ebb9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85c815a50e0e4b3688b6a8931383ebb9. Latest consensus state: current_term: 1 leader_uuid: "85c815a50e0e4b3688b6a8931383ebb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c815a50e0e4b3688b6a8931383ebb9" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 39969 } } }
I20260929 19:07:13.053972 30975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85c815a50e0e4b3688b6a8931383ebb9 [sys.catalog]: This master's current role is: LEADER
I20260929 19:07:13.054128 30972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:13.052006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51847 (local address 127.8.158.126:39969)
0929 19:07:13.052212 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:13.052219 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:13.052242 (+ 23us) server_negotiation.cc:407] Connection header received
0929 19:07:13.052332 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:13.052337 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:13.052361 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:13.052529 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:13.053575 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:13.053579 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:13.053579 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:07:13.053583 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:13.053630 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:07:13.053646 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:13.053708 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:13.053833 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 19:07:13.053874 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":122,"threads_started":1}
I20260929 19:07:13.054160 30982 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:07:13.055096 8825 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:07:13.055214 30982 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:07:13.056433 30982 catalog_manager.cc:1347] Generated new cluster ID: 8d2ee45f246d4c7fbe61a506d7ba082b
I20260929 19:07:13.056481 30982 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:07:13.057596 30972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:13.056620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35983 (local address 127.8.158.126:39969)
0929 19:07:13.056650 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:13.056652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:13.056770 (+ 118us) server_negotiation.cc:407] Connection header received
0929 19:07:13.056810 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:13.056811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:13.056815 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:13.056865 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:13.057405 (+ 540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:13.057406 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:13.057406 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:07:13.057410 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:13.057422 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:07:13.057425 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:13.057476 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:13.057530 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 19:07:13.057541 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260929 19:07:13.085925 30982 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:07:13.086087 30982 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:07:13.097285 30982 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85c815a50e0e4b3688b6a8931383ebb9: Generated new TSK 0
I20260929 19:07:13.097443 30982 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:07:13.100756 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 30900
2026-09-29T19:07:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1304 ms)
[----------] 18 tests from AuthTokenIssuingTest (23563 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:07:13 dist-test-slave-nmsz krb5kdc[30999](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:07:13 dist-test-slave-nmsz krb5kdc[30999](info): set up 2 sockets
Sep 29 19:07:13 dist-test-slave-nmsz krb5kdc[30999](info): commencing operation
krb5kdc: starting...
W20260929 19:07:14.131062 8825 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Sep 29 19:07:14 dist-test-slave-nmsz krb5kdc[30999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:07:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:07:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.158.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
I20260929 19:07:14.247040 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:35661
--webserver_interface=127.8.158.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:45475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:35661,127.8.158.125:34223,127.8.158.124:37703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:07:14.323894 31012 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 19:07:14.324117 31012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:07:14.324136 31012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:07:14.324156 31012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:07:14.325286 31012 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:07:14.325327 31012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:07:14.325341 31012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:07:14.325354 31012 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:07:14.325368 31012 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:07:14.326810 31012 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:45475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:35661,127.8.158.125:34223,127.8.158.124:37703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:35661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.158.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:07:14.327039 31012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:07:14.327255 31012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:07:14.331904 31017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:07:14.331880 31020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:07:14.331893 31018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:07:14.331976 31012 server_base.cc:1034] running on GCE node
I20260929 19:07:14.332229 31012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:07:14.332506 31012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:07:14.333659 31012 hybrid_clock.cc:648] HybridClock initialized: now 1790708834333649 us; error 27 us; skew 500 ppm
Sep 29 19:07:14 dist-test-slave-nmsz krb5kdc[30999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708834, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:07:14.336021 31012 init.cc:377] Logged in from keytab as kudu/127.8.158.126@KRBTEST.COM (short username kudu)
I20260929 19:07:14.336447 31012 webserver.cc:460] Webserver started at http://127.8.158.126:41681/ using document root <none> and password file <none>
I20260929 19:07:14.336650 31012 fs_manager.cc:359] Metadata directory not provided
I20260929 19:07:14.336712 31012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:07:14.336848 31012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:07:14.337803 31012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "a3994f4bfba74c5b97d5fb326017c61b"
format_stamp: "Formatted at 2026-09-29 19:07:14 on dist-test-slave-nmsz"
server_key: "ea05af78af12476b93d9c2b6c238bfca"
server_key_iv: "f01c1badc714d32da24c46fe9aa0e012"
server_key_version: "encryptionkey@0"
I20260929 19:07:14.338150 31012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "a3994f4bfba74c5b97d5fb326017c61b"
format_stamp: "Formatted at 2026-09-29 19:07:14 on dist-test-slave-nmsz"
server_key: "ea05af78af12476b93d9c2b6c238bfca"
server_key_iv: "f01c1badc714d32da24c46fe9aa0e012"
server_key_version: "encryptionkey@0"
I20260929 19:07:14.339489 31012 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:07:14.340229 31027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:07:14.340400 31012 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:07:14.340492 31012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "a3994f4bfba74c5b97d5fb326017c61b"
format_stamp: "Formatted at 2026-09-29 19:07:14 on dist-test-slave-nmsz"
server_key: "ea05af78af12476b93d9c2b6c238bfca"
server_key_iv: "f01c1badc714d32da24c46fe9aa0e012"
server_key_version: "encryptionkey@0"
I20260929 19:07:14.340569 31012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:07:14.347874 31012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:07:14.349746 31012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:07:14.349917 31012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:07:14.353853 31012 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.126:35661
I20260929 19:07:14.353884 31079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.126:35661 every 8 connection(s)
I20260929 19:07:14.354202 31012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:07:14.355121 31080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:07:14.356480 31080 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35661 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:07:14.360770 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 31012
I20260929 19:07:14.360852 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:07:14.361068 8825 external_mini_cluster.cc:1428] Setting key c02f855285386d41b9f3e89ce81295e0
Sep 29 19:07:14 dist-test-slave-nmsz krb5kdc[30999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708834, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:07:14.365764 31082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:14.357041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35661 (local address 127.0.0.1:33666)
0929 19:07:14.357236 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 19:07:14.357277 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 19:07:14.357743 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:07:14.358060 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:07:14.358062 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:07:14.358306 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:07:14.358503 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:07:14.358507 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.359496 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.359497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:07:14.360137 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:07:14.360139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.360647 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.360648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:07:14.360712 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:14.361212 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:07:14.361220 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:07:14.362573 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:07:14.364344 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:07:14.364350 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:07:14.364351 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:07:14.364537 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:07:14.364799 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:07:14.364800 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:07:14.364801 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:07:14.364888 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:07:14.365214 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:07:14.365218 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:07:14.365276 (+ 58us) client_negotiation.cc:769] Sending connection context
0929 19:07:14.365372 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 19:07:14.365602 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":104,"thread_start_us":81,"threads_started":1}
I20260929 19:07:14.365768 31083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:14.357097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33666 (local address 127.8.158.126:35661)
0929 19:07:14.357245 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:14.357255 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:14.357376 (+ 121us) server_negotiation.cc:407] Connection header received
0929 19:07:14.357900 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:14.357903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:14.357921 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:14.358046 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:14.358595 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.359381 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.360233 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.360535 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.360653 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:14.362674 (+ 2021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:14.362680 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:14.362681 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:07:14.362687 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:14.364255 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:14.364628 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:14.364630 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:14.364630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:14.364705 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:14.364950 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:14.364951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:14.364951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:14.365139 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:14.365220 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:14.365405 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 19:07:14.365539 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":72,"threads_started":1}
I20260929 19:07:14.366668 31080 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.158.125" port: 34223 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:07:14.366971 31082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:14.366872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.125:34223 (local address 127.0.0.1:50704)
0929 19:07:14.366913 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0929 19:07:14.366951 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.125:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260929 19:07:14.367130 31029 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.158.125:34223: connect: Connection refused (error 111)
I20260929 19:07:14.367587 31082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:14.367477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.125:34223 (local address 127.0.0.1:50710)
0929 19:07:14.367538 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0929 19:07:14.367567 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.125:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 19:07:14.367725 31080 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.158.125:34223: Network error: Client connection negotiation failed: client connection to 127.8.158.125:34223: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.8.158.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.125.keytab.
Entry for principal kudu/127.8.158.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.125.keytab.
I20260929 19:07:14.385316 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.125:34223
--webserver_interface=127.8.158.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.125.keytab
--principal=kudu/127.8.158.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:45475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:35661,127.8.158.125:34223,127.8.158.124:37703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:07:14.416950 31080 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.158.125" port: 34223 } attempt: 1
I20260929 19:07:14.417454 31082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:14.417305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.125:34223 (local address 127.0.0.1:50716)
0929 19:07:14.417369 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0929 19:07:14.417417 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.125:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 19:07:14.417806 31082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:14.417719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.125:34223 (local address 127.0.0.1:50732)
0929 19:07:14.417760 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0929 19:07:14.417787 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.125:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260929 19:07:14.417929 31080 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.158.125:34223: Network error: Client connection negotiation failed: client connection to 127.8.158.125:34223: connect: Connection refused (error 111)
W20260929 19:07:14.463044 31087 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 19:07:14.463227 31087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:07:14.463244 31087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:07:14.463285 31087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:07:14.464380 31087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:07:14.464417 31087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:07:14.464432 31087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:07:14.464466 31087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:07:14.464497 31087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:07:14.465938 31087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:45475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:35661,127.8.158.125:34223,127.8.158.124:37703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.125.keytab
--principal=kudu/127.8.158.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.125:34223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.158.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:07:14.466176 31087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:07:14.466403 31087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:07:14.470911 31092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:07:14.470932 31093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:07:14.470942 31095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:07:14.471014 31087 server_base.cc:1034] running on GCE node
I20260929 19:07:14.471364 31087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:07:14.471616 31087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:07:14.472769 31087 hybrid_clock.cc:648] HybridClock initialized: now 1790708834472760 us; error 38 us; skew 500 ppm
Sep 29 19:07:14 dist-test-slave-nmsz krb5kdc[30999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708834, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:07:14.475116 31087 init.cc:377] Logged in from keytab as kudu/127.8.158.125@KRBTEST.COM (short username kudu)
I20260929 19:07:14.475456 31087 webserver.cc:460] Webserver started at http://127.8.158.125:45881/ using document root <none> and password file <none>
I20260929 19:07:14.475629 31087 fs_manager.cc:359] Metadata directory not provided
I20260929 19:07:14.475692 31087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:07:14.475782 31087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:07:14.476650 31087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-1/data/instance:
uuid: "ca7fc4e721de4dd2996f0c5b6fad02ee"
format_stamp: "Formatted at 2026-09-29 19:07:14 on dist-test-slave-nmsz"
server_key: "b5491f6946253a2f8f1918f0740580c9"
server_key_iv: "31aecd0af83f894a1f50f9be94fe71ec"
server_key_version: "encryptionkey@0"
I20260929 19:07:14.476923 31087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-1/wal/instance:
uuid: "ca7fc4e721de4dd2996f0c5b6fad02ee"
format_stamp: "Formatted at 2026-09-29 19:07:14 on dist-test-slave-nmsz"
server_key: "b5491f6946253a2f8f1918f0740580c9"
server_key_iv: "31aecd0af83f894a1f50f9be94fe71ec"
server_key_version: "encryptionkey@0"
I20260929 19:07:14.478093 31087 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 19:07:14.478801 31102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:07:14.479228 31087 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 19:07:14.479329 31087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-1/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-1/wal
uuid: "ca7fc4e721de4dd2996f0c5b6fad02ee"
format_stamp: "Formatted at 2026-09-29 19:07:14 on dist-test-slave-nmsz"
server_key: "b5491f6946253a2f8f1918f0740580c9"
server_key_iv: "31aecd0af83f894a1f50f9be94fe71ec"
server_key_version: "encryptionkey@0"
I20260929 19:07:14.479382 31087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:07:14.486080 31080 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.158.125" port: 34223 } attempt: 2
I20260929 19:07:14.486466 31082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:14.486350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.125:34223 (local address 127.0.0.1:50748)
0929 19:07:14.486395 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0929 19:07:14.486432 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.125:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260929 19:07:14.486801 31082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:14.486718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.125:34223 (local address 127.0.0.1:50764)
0929 19:07:14.486757 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0929 19:07:14.486782 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.125:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260929 19:07:14.486901 31080 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.158.125:34223: Network error: Client connection negotiation failed: client connection to 127.8.158.125:34223: connect: Connection refused (error 111)
I20260929 19:07:14.489617 31087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:07:14.491398 31087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:07:14.491541 31087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:07:14.495180 31087 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.125:34223
I20260929 19:07:14.495208 31154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.125:34223 every 8 connection(s)
I20260929 19:07:14.495527 31087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-1/data/info.pb
I20260929 19:07:14.496414 31155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:07:14.497766 31155 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35661 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:07:14.499393 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 31087
I20260929 19:07:14.499493 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-1/wal/instance
I20260929 19:07:14.499645 8825 external_mini_cluster.cc:1428] Setting key 9f6335436c0f1005a53332da5e2faae3
Sep 29 19:07:14 dist-test-slave-nmsz krb5kdc[30999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708834, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.125@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:07:14.506537 31083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:14.498372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33668 (local address 127.8.158.126:35661)
0929 19:07:14.498415 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:14.498416 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:14.498622 (+ 206us) server_negotiation.cc:407] Connection header received
0929 19:07:14.499116 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:14.499118 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:14.499132 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:14.499182 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:14.499859 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.500330 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.501277 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.501473 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.501545 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:14.503636 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:14.503638 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:14.503638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:07:14.503647 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:14.505240 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:14.505592 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:14.505593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:14.505593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:14.505658 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:14.505881 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:14.505882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:14.505883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:14.506022 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:14.506099 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:14.506340 (+ 241us) server_negotiation.cc:299] Negotiation successful
0929 19:07:14.506417 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 19:07:14.506559 31157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:14.498365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35661 (local address 127.0.0.1:33668)
0929 19:07:14.498514 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 19:07:14.498546 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 19:07:14.499008 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:07:14.499254 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:07:14.499258 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:07:14.499505 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:07:14.499730 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:07:14.499735 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.500414 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.500415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:07:14.501173 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:07:14.501175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.501564 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.501565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:07:14.501698 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:14.502183 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:07:14.502191 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:07:14.503550 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:07:14.505325 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:07:14.505331 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:07:14.505333 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:07:14.505523 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:07:14.505729 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:07:14.505730 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:07:14.505730 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:07:14.505822 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:07:14.506087 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:07:14.506091 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:07:14.506190 (+ 99us) client_negotiation.cc:769] Sending connection context
0929 19:07:14.506263 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 19:07:14.506420 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":63,"threads_started":1}
I20260929 19:07:14.507063 31155 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.158.125" port: 34223 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.8.158.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.124@KRBTEST.COM" created.
Sep 29 19:07:14 dist-test-slave-nmsz krb5kdc[30999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708834, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.125@KRBTEST.COM for kudu/127.8.158.125@KRBTEST.COM
I20260929 19:07:14.514066 31157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:14.507290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.125:34223 (local address 127.0.0.1:50772)
0929 19:07:14.507334 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0929 19:07:14.507345 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:07:14.507388 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:07:14.507618 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:07:14.507619 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:07:14.507769 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:07:14.507826 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:07:14.507827 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.508587 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.508588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:07:14.509071 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:07:14.509072 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.509521 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.509522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:07:14.509583 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:14.510012 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:07:14.510017 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:07:14.511476 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:07:14.513103 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:07:14.513104 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:07:14.513104 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:07:14.513258 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:07:14.513465 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:07:14.513466 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:07:14.513466 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:07:14.513517 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:07:14.513769 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:07:14.513770 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:07:14.513804 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 19:07:14.513869 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 19:07:14.513962 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260929 19:07:14.514205 31159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:14.507342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50772 (local address 127.8.158.125:34223)
0929 19:07:14.507476 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:14.507480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:14.507498 (+ 18us) server_negotiation.cc:407] Connection header received
0929 19:07:14.507520 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:14.507522 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:14.507538 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:14.507596 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:14.507888 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.508508 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.509160 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.509440 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.509522 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:14.511564 (+ 2042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:14.511571 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:14.511572 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:07:14.511577 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:14.513006 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:14.513350 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:14.513353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:14.513353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:14.513404 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:14.513583 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:14.513583 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:14.513584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:14.513704 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:14.513795 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:14.513904 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 19:07:14.514011 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":109,"thread_start_us":89,"threads_started":1}
I20260929 19:07:14.514664 31155 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.158.124" port: 37703 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:07:14.514967 31157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:14.514864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.124:37703 (local address 127.0.0.1:33022)
0929 19:07:14.514904 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0929 19:07:14.514945 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.124:37703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260929 19:07:14.515094 31105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.158.124:37703: connect: Connection refused (error 111)
I20260929 19:07:14.515446 31157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:14.515359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.124:37703 (local address 127.0.0.1:33028)
0929 19:07:14.515404 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0929 19:07:14.515426 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.124:37703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 19:07:14.515540 31155 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.158.124:37703: Network error: Client connection negotiation failed: client connection to 127.8.158.124:37703: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.124.keytab.
Entry for principal kudu/127.8.158.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.124.keytab.
I20260929 19:07:14.524314 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.124:37703
--webserver_interface=127.8.158.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.124.keytab
--principal=kudu/127.8.158.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:45475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:35661,127.8.158.125:34223,127.8.158.124:37703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:07:14.564735 31155 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.158.124" port: 37703 } attempt: 1
I20260929 19:07:14.565152 31157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:14.565036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.124:37703 (local address 127.0.0.1:33032)
0929 19:07:14.565092 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0929 19:07:14.565126 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.124:37703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 19:07:14.565469 31157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:14.565380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.124:37703 (local address 127.0.0.1:33036)
0929 19:07:14.565422 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0929 19:07:14.565445 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.124:37703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260929 19:07:14.565555 31155 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.158.124:37703: Network error: Client connection negotiation failed: client connection to 127.8.158.124:37703: connect: Connection refused (error 111)
I20260929 19:07:14.578068 31080 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.158.125" port: 34223 } attempt: 3
Sep 29 19:07:14 dist-test-slave-nmsz krb5kdc[30999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708834, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for kudu/127.8.158.125@KRBTEST.COM
I20260929 19:07:14.585052 31159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:14.578392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50776 (local address 127.8.158.125:34223)
0929 19:07:14.578428 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:14.578430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:14.578469 (+ 39us) server_negotiation.cc:407] Connection header received
0929 19:07:14.578549 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:14.578549 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:14.578558 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:14.578601 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:14.578937 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.579411 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.580374 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.580575 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.580624 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:14.582505 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:14.582506 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:14.582506 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:07:14.582514 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:14.583987 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:14.584330 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:14.584331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:14.584332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:14.584387 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:14.584572 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:14.584573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:14.584573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:14.584674 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:14.584754 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:14.584896 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 19:07:14.584964 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 19:07:14.585054 31082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:14.578352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.125:34223 (local address 127.0.0.1:50776)
0929 19:07:14.578406 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0929 19:07:14.578419 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 19:07:14.578489 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:07:14.578616 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:07:14.578617 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:07:14.578794 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:07:14.578868 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:07:14.578871 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.579485 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.579486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:07:14.580300 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:07:14.580303 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.580643 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.580643 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:07:14.580722 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:14.581133 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:07:14.581138 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:07:14.582427 (+ 1289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:07:14.584085 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:07:14.584086 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:07:14.584087 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:07:14.584250 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:07:14.584450 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:07:14.584450 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:07:14.584451 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:07:14.584507 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:07:14.584747 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:07:14.584748 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:07:14.584790 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 19:07:14.584873 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 19:07:14.584964 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260929 19:07:14.585529 31080 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.158.124" port: 37703 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:07:14.585878 31082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:14.585772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.124:37703 (local address 127.0.0.1:33040)
0929 19:07:14.585829 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0929 19:07:14.585854 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.124:37703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 19:07:14.586252 31082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:14.586173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.124:37703 (local address 127.0.0.1:33056)
0929 19:07:14.586213 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0929 19:07:14.586234 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.124:37703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 19:07:14.586367 31080 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.158.124:37703: Network error: Client connection negotiation failed: client connection to 127.8.158.124:37703: connect: Connection refused (error 111)
W20260929 19:07:14.603334 31162 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 19:07:14.603511 31162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:07:14.603528 31162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:07:14.603547 31162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:07:14.604668 31162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:07:14.604707 31162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:07:14.604720 31162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:07:14.604733 31162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:07:14.604784 31162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:07:14.606215 31162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:45475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:35661,127.8.158.125:34223,127.8.158.124:37703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.124.keytab
--principal=kudu/127.8.158.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.124:37703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.158.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:07:14.606454 31162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:07:14.606668 31162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:07:14.610984 31168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:07:14.611030 31170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:07:14.611029 31167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:07:14.611470 31162 server_base.cc:1034] running on GCE node
I20260929 19:07:14.611668 31162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:07:14.611878 31162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:07:14.613015 31162 hybrid_clock.cc:648] HybridClock initialized: now 1790708834613002 us; error 26 us; skew 500 ppm
Sep 29 19:07:14 dist-test-slave-nmsz krb5kdc[30999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708834, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:07:14.615314 31162 init.cc:377] Logged in from keytab as kudu/127.8.158.124@KRBTEST.COM (short username kudu)
I20260929 19:07:14.615700 31162 webserver.cc:460] Webserver started at http://127.8.158.124:34991/ using document root <none> and password file <none>
I20260929 19:07:14.615900 31162 fs_manager.cc:359] Metadata directory not provided
I20260929 19:07:14.615943 31162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:07:14.616079 31162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:07:14.617038 31162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-2/data/instance:
uuid: "8a19f00263154bdfaf93f0f2dd4f60bb"
format_stamp: "Formatted at 2026-09-29 19:07:14 on dist-test-slave-nmsz"
server_key: "c1309071b6e173782aab15b2d5d1268b"
server_key_iv: "5490f8900598fe6b3bbbfcb3595e3a2b"
server_key_version: "encryptionkey@0"
I20260929 19:07:14.617367 31162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-2/wal/instance:
uuid: "8a19f00263154bdfaf93f0f2dd4f60bb"
format_stamp: "Formatted at 2026-09-29 19:07:14 on dist-test-slave-nmsz"
server_key: "c1309071b6e173782aab15b2d5d1268b"
server_key_iv: "5490f8900598fe6b3bbbfcb3595e3a2b"
server_key_version: "encryptionkey@0"
I20260929 19:07:14.617502 31080 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.158.124" port: 37703 } attempt: 1
I20260929 19:07:14.617837 31082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:14.617717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.124:37703 (local address 127.0.0.1:33064)
0929 19:07:14.617769 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0929 19:07:14.617799 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.124:37703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260929 19:07:14.618158 31082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:14.618081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.124:37703 (local address 127.0.0.1:33068)
0929 19:07:14.618122 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0929 19:07:14.618144 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.124:37703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 19:07:14.618242 31080 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.158.124:37703: Network error: Client connection negotiation failed: client connection to 127.8.158.124:37703: connect: Connection refused (error 111)
I20260929 19:07:14.618927 31162 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:07:14.619613 31177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:07:14.619761 31162 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:07:14.619849 31162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-2/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-2/wal
uuid: "8a19f00263154bdfaf93f0f2dd4f60bb"
format_stamp: "Formatted at 2026-09-29 19:07:14 on dist-test-slave-nmsz"
server_key: "c1309071b6e173782aab15b2d5d1268b"
server_key_iv: "5490f8900598fe6b3bbbfcb3595e3a2b"
server_key_version: "encryptionkey@0"
I20260929 19:07:14.619923 31162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:07:14.633692 31155 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.158.124" port: 37703 } attempt: 2
I20260929 19:07:14.634044 31157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:14.633932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.124:37703 (local address 127.0.0.1:33082)
0929 19:07:14.633985 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0929 19:07:14.634017 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.124:37703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 19:07:14.634371 31157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:14.634286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.124:37703 (local address 127.0.0.1:33084)
0929 19:07:14.634334 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 19:07:14.634356 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.124:37703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260929 19:07:14.634477 31155 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.158.124:37703: Network error: Client connection negotiation failed: client connection to 127.8.158.124:37703: connect: Connection refused (error 111)
I20260929 19:07:14.636380 31162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:07:14.638180 31162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:07:14.638321 31162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:07:14.642050 31162 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.124:37703
I20260929 19:07:14.642099 31229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.124:37703 every 8 connection(s)
I20260929 19:07:14.642450 31162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-2/data/info.pb
I20260929 19:07:14.643417 31230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:07:14.644853 31230 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35661 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:07:14.648451 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 31162
I20260929 19:07:14.648531 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708573736807-8825-0/minicluster-data/master-2/wal/instance
I20260929 19:07:14.648710 8825 external_mini_cluster.cc:1428] Setting key eb1aba5b9ccb595200813f98fffb0ca1
Sep 29 19:07:14 dist-test-slave-nmsz krb5kdc[30999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708834, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.124@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:07:14.654064 31083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:14.645474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33678 (local address 127.8.158.126:35661)
0929 19:07:14.645512 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:14.645514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:14.645752 (+ 238us) server_negotiation.cc:407] Connection header received
0929 19:07:14.646225 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:14.646226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:14.646239 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:14.646285 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:14.646876 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.647335 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.648278 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.648579 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.648659 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:14.651180 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:14.651181 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:14.651181 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:07:14.651188 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:14.652698 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:14.653128 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:14.653129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:14.653130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:14.653193 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:14.653448 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:14.653449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:14.653449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:14.653589 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:14.653661 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:14.653888 (+ 227us) server_negotiation.cc:299] Negotiation successful
0929 19:07:14.653974 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 19:07:14.654081 31232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:14.645475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:35661 (local address 127.0.0.1:33678)
0929 19:07:14.645644 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 19:07:14.645674 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 19:07:14.646117 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:07:14.646363 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:07:14.646366 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:07:14.646605 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:07:14.646781 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:07:14.646785 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.647431 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.647432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:07:14.648189 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:07:14.648191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.648681 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.648682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:07:14.648834 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:14.649347 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:07:14.649357 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:07:14.651103 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:07:14.652762 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:07:14.652768 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:07:14.652770 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:07:14.653033 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:07:14.653274 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:07:14.653275 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:07:14.653275 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:07:14.653374 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:07:14.653664 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:07:14.653667 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:07:14.653764 (+ 97us) client_negotiation.cc:769] Sending connection context
0929 19:07:14.653840 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 19:07:14.653940 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":86,"threads_started":1}
I20260929 19:07:14.654592 31230 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.158.125" port: 34223 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 19:07:14 dist-test-slave-nmsz krb5kdc[30999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.158.124@KRBTEST.COM
I20260929 19:07:14.658143 31234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:14.649074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33094 (local address 127.8.158.124:37703)
0929 19:07:14.649235 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:14.649240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:14.649397 (+ 157us) server_negotiation.cc:407] Connection header received
0929 19:07:14.649454 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:14.649456 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:14.649475 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:14.649533 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:14.650237 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.651275 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.652361 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.652639 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.652734 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:14.655401 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:14.655408 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:14.655408 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:07:14.655414 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:14.657007 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:14.657340 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:14.657343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:14.657343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:14.657401 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:14.657589 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:14.657590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:14.657591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:14.657729 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:14.657873 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:14.657960 (+ 87us) server_negotiation.cc:299] Negotiation successful
0929 19:07:14.658033 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":98,"threads_started":1}
Sep 29 19:07:14 dist-test-slave-nmsz krb5kdc[30999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708834, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.124@KRBTEST.COM for kudu/127.8.158.125@KRBTEST.COM
I20260929 19:07:14.661329 31159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:14.654879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50778 (local address 127.8.158.125:34223)
0929 19:07:14.654918 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:14.654920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:14.654928 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:07:14.654989 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:14.654990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:14.654999 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:14.655057 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:14.655343 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.655902 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.656868 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.657061 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.657154 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:14.658939 (+ 1785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:14.658940 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:14.658941 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:07:14.658948 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:14.660292 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:14.660626 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:14.660627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:14.660628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:14.660681 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:14.660865 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:14.660866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:14.660866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:14.660967 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:14.661037 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:14.661176 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 19:07:14.661243 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 19:07:14.661327 31232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:14.654821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.125:34223 (local address 127.0.0.1:50778)
0929 19:07:14.654869 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 19:07:14.654881 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:07:14.654931 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:07:14.655065 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:07:14.655066 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:07:14.655217 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:07:14.655269 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:07:14.655271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.655985 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.655986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:07:14.656768 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:07:14.656770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.657157 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.657158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:07:14.657228 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:14.657588 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:07:14.657593 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:07:14.658839 (+ 1246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:07:14.660388 (+ 1549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:07:14.660389 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:07:14.660389 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:07:14.660548 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:07:14.660748 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:07:14.660749 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:07:14.660749 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:07:14.660802 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:07:14.661030 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:07:14.661031 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:07:14.661070 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 19:07:14.661152 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 19:07:14.661234 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 19:07:14.661718 31230 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.158.124" port: 37703 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 19:07:14 dist-test-slave-nmsz krb5kdc[30999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
Sep 29 19:07:14 dist-test-slave-nmsz krb5kdc[30999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708834, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.124@KRBTEST.COM for kudu/127.8.158.124@KRBTEST.COM
I20260929 19:07:14.665959 31083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:14.658896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33688 (local address 127.8.158.126:35661)
0929 19:07:14.659162 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:14.659164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:14.659170 (+ 6us) server_negotiation.cc:407] Connection header received
0929 19:07:14.659185 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:14.659186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:14.659193 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:14.659257 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:14.659691 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.660122 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.660763 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.660961 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.661039 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:14.663457 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:14.663458 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:14.663459 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:07:14.663464 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:14.664928 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:14.665219 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:14.665220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:14.665220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:14.665276 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:14.665446 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:14.665447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:14.665447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:14.665572 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:14.665734 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:14.665784 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 19:07:14.665864 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249}
I20260929 19:07:14.668203 31232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:14.661943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.124:37703 (local address 127.0.0.1:33110)
0929 19:07:14.661984 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0929 19:07:14.661994 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:07:14.662027 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:07:14.662159 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:07:14.662159 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:07:14.662257 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:07:14.662282 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:07:14.662282 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.662977 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.662978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:07:14.663670 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:07:14.663673 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.664018 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.664019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:07:14.664067 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:14.664564 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:07:14.664568 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:07:14.665711 (+ 1143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:07:14.667267 (+ 1556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:07:14.667267 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:07:14.667268 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:07:14.667422 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:07:14.667615 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:07:14.667615 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:07:14.667616 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:07:14.667666 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:07:14.667895 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:07:14.667895 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:07:14.667937 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 19:07:14.668015 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 19:07:14.668101 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260929 19:07:14.668258 31234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:14.662005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33110 (local address 127.8.158.124:37703)
0929 19:07:14.662042 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:14.662044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:14.662052 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:07:14.662073 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:14.662074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:14.662085 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:14.662158 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:14.662374 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.662905 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.663739 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.663932 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.664017 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:14.665797 (+ 1780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:14.665799 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:14.665799 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:07:14.665805 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:14.667176 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:14.667500 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:14.667501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:14.667502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:14.667548 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:14.667740 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:14.667741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:14.667741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:14.667836 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:14.667902 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:14.668037 (+ 135us) server_negotiation.cc:299] Negotiation successful
0929 19:07:14.668148 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":19}
I20260929 19:07:14.669631 31230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a19f00263154bdfaf93f0f2dd4f60bb: Bootstrap starting.
I20260929 19:07:14.670662 31230 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a19f00263154bdfaf93f0f2dd4f60bb: Neither blocks nor log segments found. Creating new log.
I20260929 19:07:14.670948 31230 log.cc:824] T 00000000000000000000000000000000 P 8a19f00263154bdfaf93f0f2dd4f60bb: Log is configured to *not* fsync() on all Append() calls
Sep 29 19:07:14 dist-test-slave-nmsz krb5kdc[30999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.158.125@KRBTEST.COM
I20260929 19:07:14.671589 31230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a19f00263154bdfaf93f0f2dd4f60bb: No bootstrap required, opened a new log
I20260929 19:07:14.673182 31230 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a19f00263154bdfaf93f0f2dd4f60bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3994f4bfba74c5b97d5fb326017c61b" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35661 } } peers { permanent_uuid: "ca7fc4e721de4dd2996f0c5b6fad02ee" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 34223 } } peers { permanent_uuid: "8a19f00263154bdfaf93f0f2dd4f60bb" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 37703 } }
I20260929 19:07:14.673317 31230 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a19f00263154bdfaf93f0f2dd4f60bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a19f00263154bdfaf93f0f2dd4f60bb, State: Initialized, Role: FOLLOWER
I20260929 19:07:14.673426 31230 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a19f00263154bdfaf93f0f2dd4f60bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3994f4bfba74c5b97d5fb326017c61b" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35661 } } peers { permanent_uuid: "ca7fc4e721de4dd2996f0c5b6fad02ee" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 34223 } } peers { permanent_uuid: "8a19f00263154bdfaf93f0f2dd4f60bb" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 37703 } }
I20260929 19:07:14.673674 31236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a19f00263154bdfaf93f0f2dd4f60bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3994f4bfba74c5b97d5fb326017c61b" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35661 } } peers { permanent_uuid: "ca7fc4e721de4dd2996f0c5b6fad02ee" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 34223 } } peers { permanent_uuid: "8a19f00263154bdfaf93f0f2dd4f60bb" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 37703 } } }
I20260929 19:07:14.673724 31230 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a19f00263154bdfaf93f0f2dd4f60bb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:07:14.673825 31159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:14.666923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50782 (local address 127.8.158.125:34223)
0929 19:07:14.666956 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:14.666958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:14.666996 (+ 38us) server_negotiation.cc:407] Connection header received
0929 19:07:14.667063 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:14.667064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:14.667073 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:14.667120 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:14.667614 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.668037 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.668872 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.669058 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.669111 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:14.671480 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:14.671482 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:14.671482 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:07:14.671488 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:14.672805 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:14.673122 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:14.673123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:14.673124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:14.673178 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:14.673362 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:14.673363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:14.673363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:14.673467 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:14.673610 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:14.673649 (+ 39us) server_negotiation.cc:299] Negotiation successful
0929 19:07:14.673714 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 19:07:14.673796 31236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a19f00263154bdfaf93f0f2dd4f60bb [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:07:14.675076 8825 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260929 19:07:14.675674 31247 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8a19f00263154bdfaf93f0f2dd4f60bb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 19:07:14.675729 31247 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:07:14.676251 31054 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "a3994f4bfba74c5b97d5fb326017c61b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33688
I20260929 19:07:14.676299 31162 master_runner.cc:186] Error getting master registration for 127.8.158.126:35661: OK, instance_id { permanent_uuid: "a3994f4bfba74c5b97d5fb326017c61b" instance_seqno: 1790708834352780 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 19:07:14.679087 31162 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 19:07:14.693373 31080 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.158.124" port: 37703 } attempt: 2
Sep 29 19:07:14 dist-test-slave-nmsz krb5kdc[30999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708834, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for kudu/127.8.158.124@KRBTEST.COM
I20260929 19:07:14.699993 31082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:14.693625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.124:37703 (local address 127.0.0.1:33112)
0929 19:07:14.693662 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0929 19:07:14.693674 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:07:14.693732 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:07:14.693880 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:07:14.693881 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:07:14.694030 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:07:14.694066 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:07:14.694067 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.694743 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.694744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:07:14.695229 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:07:14.695231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.695580 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.695581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:07:14.695636 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:14.696030 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:07:14.696035 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:07:14.697170 (+ 1135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:07:14.698926 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:07:14.698927 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:07:14.698927 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:07:14.699133 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:07:14.699327 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:07:14.699328 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:07:14.699328 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:07:14.699389 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:07:14.699702 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:07:14.699703 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:07:14.699743 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 19:07:14.699810 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 19:07:14.699895 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260929 19:07:14.700011 31234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:14.693685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33112 (local address 127.8.158.124:37703)
0929 19:07:14.693731 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:14.693733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:14.693742 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:07:14.693801 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:14.693803 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:14.693812 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:14.693877 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:14.694142 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.694668 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.695312 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.695511 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.695582 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:14.697259 (+ 1677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:14.697260 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:14.697260 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:07:14.697267 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:14.698850 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:14.699205 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:14.699206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:14.699206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:14.699262 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:14.699475 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:14.699476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:14.699476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:14.699634 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:14.699703 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:14.699852 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 19:07:14.699924 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 19:07:14.701455 31080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3994f4bfba74c5b97d5fb326017c61b: Bootstrap starting.
I20260929 19:07:14.702144 31080 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3994f4bfba74c5b97d5fb326017c61b: Neither blocks nor log segments found. Creating new log.
I20260929 19:07:14.702466 31080 log.cc:824] T 00000000000000000000000000000000 P a3994f4bfba74c5b97d5fb326017c61b: Log is configured to *not* fsync() on all Append() calls
I20260929 19:07:14.703200 31080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3994f4bfba74c5b97d5fb326017c61b: No bootstrap required, opened a new log
I20260929 19:07:14.704492 31080 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3994f4bfba74c5b97d5fb326017c61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3994f4bfba74c5b97d5fb326017c61b" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35661 } } peers { permanent_uuid: "ca7fc4e721de4dd2996f0c5b6fad02ee" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 34223 } } peers { permanent_uuid: "8a19f00263154bdfaf93f0f2dd4f60bb" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 37703 } }
I20260929 19:07:14.704622 31080 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3994f4bfba74c5b97d5fb326017c61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3994f4bfba74c5b97d5fb326017c61b, State: Initialized, Role: FOLLOWER
I20260929 19:07:14.704694 31080 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3994f4bfba74c5b97d5fb326017c61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3994f4bfba74c5b97d5fb326017c61b" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35661 } } peers { permanent_uuid: "ca7fc4e721de4dd2996f0c5b6fad02ee" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 34223 } } peers { permanent_uuid: "8a19f00263154bdfaf93f0f2dd4f60bb" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 37703 } }
I20260929 19:07:14.704986 31080 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3994f4bfba74c5b97d5fb326017c61b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:07:14.705013 31054 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3994f4bfba74c5b97d5fb326017c61b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 19:07:14.705173 31054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3994f4bfba74c5b97d5fb326017c61b [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:07:14.705907 31054 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3994f4bfba74c5b97d5fb326017c61b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3994f4bfba74c5b97d5fb326017c61b" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35661 } } peers { permanent_uuid: "ca7fc4e721de4dd2996f0c5b6fad02ee" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 34223 } } peers { permanent_uuid: "8a19f00263154bdfaf93f0f2dd4f60bb" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 37703 } }
I20260929 19:07:14.706139 31054 leader_election.cc:288] T 00000000000000000000000000000000 P a3994f4bfba74c5b97d5fb326017c61b [CANDIDATE]: Term 1 election: Requested vote from peers ca7fc4e721de4dd2996f0c5b6fad02ee (127.8.158.125:34223), 8a19f00263154bdfaf93f0f2dd4f60bb (127.8.158.124:37703)
I20260929 19:07:14.706604 31204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a3994f4bfba74c5b97d5fb326017c61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8a19f00263154bdfaf93f0f2dd4f60bb"
I20260929 19:07:14.706707 31204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a19f00263154bdfaf93f0f2dd4f60bb [term 0 FOLLOWER]: Advancing to term 1
W20260929 19:07:14.706847 31261 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a3994f4bfba74c5b97d5fb326017c61b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 19:07:14.706889 31261 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:07:14.704972 31250 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3994f4bfba74c5b97d5fb326017c61b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3994f4bfba74c5b97d5fb326017c61b" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35661 } } peers { permanent_uuid: "ca7fc4e721de4dd2996f0c5b6fad02ee" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 34223 } } peers { permanent_uuid: "8a19f00263154bdfaf93f0f2dd4f60bb" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 37703 } } }
I20260929 19:07:14.706987 31250 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3994f4bfba74c5b97d5fb326017c61b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:07:14.707506 31204 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8a19f00263154bdfaf93f0f2dd4f60bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3994f4bfba74c5b97d5fb326017c61b in term 1.
I20260929 19:07:14.707772 31030 leader_election.cc:302] T 00000000000000000000000000000000 P a3994f4bfba74c5b97d5fb326017c61b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a19f00263154bdfaf93f0f2dd4f60bb, a3994f4bfba74c5b97d5fb326017c61b; no voters:
I20260929 19:07:14.707906 31250 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3994f4bfba74c5b97d5fb326017c61b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:07:14.708057 31250 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3994f4bfba74c5b97d5fb326017c61b [term 1 LEADER]: Becoming Leader. State: Replica: a3994f4bfba74c5b97d5fb326017c61b, State: Running, Role: LEADER
I20260929 19:07:14.708127 31250 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3994f4bfba74c5b97d5fb326017c61b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3994f4bfba74c5b97d5fb326017c61b" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35661 } } peers { permanent_uuid: "ca7fc4e721de4dd2996f0c5b6fad02ee" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 34223 } } peers { permanent_uuid: "8a19f00263154bdfaf93f0f2dd4f60bb" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 37703 } }
I20260929 19:07:14.708509 31250 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3994f4bfba74c5b97d5fb326017c61b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3994f4bfba74c5b97d5fb326017c61b. Latest consensus state: current_term: 1 leader_uuid: "a3994f4bfba74c5b97d5fb326017c61b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3994f4bfba74c5b97d5fb326017c61b" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35661 } } peers { permanent_uuid: "ca7fc4e721de4dd2996f0c5b6fad02ee" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 34223 } } peers { permanent_uuid: "8a19f00263154bdfaf93f0f2dd4f60bb" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 37703 } } }
I20260929 19:07:14.708578 31250 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3994f4bfba74c5b97d5fb326017c61b [sys.catalog]: This master's current role is: LEADER
I20260929 19:07:14.708896 31265 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:07:14.709126 31265 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:07:14.708222 31012 master_runner.cc:186] Error getting master registration for 127.8.158.125:34223: OK, instance_id { permanent_uuid: "ca7fc4e721de4dd2996f0c5b6fad02ee" instance_seqno: 1790708834494130 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 19:07:14.710391 31012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 19:07:14.711493 31204 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8a19f00263154bdfaf93f0f2dd4f60bb [term 1 FOLLOWER]: Refusing update from remote peer a3994f4bfba74c5b97d5fb326017c61b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:07:14.711807 31263 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a3994f4bfba74c5b97d5fb326017c61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a19f00263154bdfaf93f0f2dd4f60bb" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 37703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:07:14.713048 31236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a19f00263154bdfaf93f0f2dd4f60bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3994f4bfba74c5b97d5fb326017c61b. Latest consensus state: current_term: 1 leader_uuid: "a3994f4bfba74c5b97d5fb326017c61b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3994f4bfba74c5b97d5fb326017c61b" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35661 } } peers { permanent_uuid: "ca7fc4e721de4dd2996f0c5b6fad02ee" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 34223 } } peers { permanent_uuid: "8a19f00263154bdfaf93f0f2dd4f60bb" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 37703 } } }
I20260929 19:07:14.713128 31236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a19f00263154bdfaf93f0f2dd4f60bb [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:07:14.714386 31263 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3994f4bfba74c5b97d5fb326017c61b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a3994f4bfba74c5b97d5fb326017c61b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3994f4bfba74c5b97d5fb326017c61b" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35661 } } peers { permanent_uuid: "ca7fc4e721de4dd2996f0c5b6fad02ee" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 34223 } } peers { permanent_uuid: "8a19f00263154bdfaf93f0f2dd4f60bb" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 37703 } } }
I20260929 19:07:14.714484 31263 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3994f4bfba74c5b97d5fb326017c61b [sys.catalog]: This master's current role is: LEADER
I20260929 19:07:14.714978 31236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a19f00263154bdfaf93f0f2dd4f60bb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a3994f4bfba74c5b97d5fb326017c61b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3994f4bfba74c5b97d5fb326017c61b" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35661 } } peers { permanent_uuid: "ca7fc4e721de4dd2996f0c5b6fad02ee" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 34223 } } peers { permanent_uuid: "8a19f00263154bdfaf93f0f2dd4f60bb" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 37703 } } }
I20260929 19:07:14.715049 31236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a19f00263154bdfaf93f0f2dd4f60bb [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:07:14.715291 31266 mvcc.cc:205] Tried to move back new op lower bound from 7334743386976063488 to 7334743386965176320. Current Snapshot: MvccSnapshot[applied={T|T < 7334743386976063488}]
I20260929 19:07:14.715412 31272 mvcc.cc:205] Tried to move back new op lower bound from 7334743386976063488 to 7334743386965176320. Current Snapshot: MvccSnapshot[applied={T|T < 7334743386976063488}]
I20260929 19:07:14.722750 31265 catalog_manager.cc:1347] Generated new cluster ID: 4ae563ed44ef4cc597a8413ab3570bb0
I20260929 19:07:14.722800 31265 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:07:14.725603 31155 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.158.124" port: 37703 } attempt: 3
I20260929 19:07:14.727102 31083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:14.711985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33700 (local address 127.8.158.126:35661)
0929 19:07:14.712051 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:14.712053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:14.715676 (+ 3623us) server_negotiation.cc:407] Connection header received
0929 19:07:14.715767 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:14.715768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:14.715777 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:14.715804 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:14.717401 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.718133 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.720093 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.720379 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.720416 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:14.721935 (+ 1519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:14.721936 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:14.721936 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:07:14.721943 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:14.723696 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:14.724532 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:14.724533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:14.724533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:14.724831 (+ 298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:14.726556 (+ 1725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:14.726557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:14.726557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:14.726677 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:14.726777 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:14.726957 (+ 180us) server_negotiation.cc:299] Negotiation successful
0929 19:07:14.727021 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 19:07:14.728449 31234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:14.711495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33114 (local address 127.8.158.124:37703)
0929 19:07:14.711529 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:14.711531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:14.713673 (+ 2142us) server_negotiation.cc:407] Connection header received
0929 19:07:14.713828 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:14.713829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:14.713857 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:14.713891 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:14.715386 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.716092 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.717887 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.718194 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.718235 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:14.723878 (+ 5643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:14.723879 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:14.723880 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:07:14.723887 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:14.725434 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:14.726757 (+ 1323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:14.726758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:14.726759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:14.726826 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:14.727511 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:14.727513 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:14.727513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:14.727636 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:14.727710 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:14.728292 (+ 582us) server_negotiation.cc:299] Negotiation successful
0929 19:07:14.728364 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Sep 29 19:07:14 dist-test-slave-nmsz krb5kdc[30999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708834, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.125@KRBTEST.COM for kudu/127.8.158.124@KRBTEST.COM
I20260929 19:07:14.737157 31157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:14.725839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.124:37703 (local address 127.0.0.1:33122)
0929 19:07:14.725876 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0929 19:07:14.725887 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:07:14.725925 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:07:14.728658 (+ 2733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:07:14.728659 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:07:14.728831 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:07:14.728868 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:07:14.728869 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.729775 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.729775 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:07:14.730580 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:07:14.730582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.731115 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.731116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:07:14.731186 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:14.731565 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:07:14.731569 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:07:14.733040 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:07:14.735806 (+ 2766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:07:14.735807 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:07:14.735808 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:07:14.736040 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:07:14.736374 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:07:14.736375 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:07:14.736375 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:07:14.736444 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:07:14.736853 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:07:14.736853 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:07:14.736908 (+ 55us) client_negotiation.cc:769] Sending connection context
0929 19:07:14.736974 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 19:07:14.737061 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260929 19:07:14.737974 31234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:14.725943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33122 (local address 127.8.158.124:37703)
0929 19:07:14.728515 (+ 2572us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:14.728517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:14.728524 (+ 7us) server_negotiation.cc:407] Connection header received
0929 19:07:14.728541 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:14.728542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:14.728550 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:14.728601 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:14.728967 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.729668 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.730686 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.730998 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.731053 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:14.733145 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:14.733146 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:14.733147 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:07:14.733153 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:14.735676 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:14.736137 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:14.736168 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:14.736169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:14.736248 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:14.736517 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:14.736518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:14.736518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:14.736723 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:14.737622 (+ 899us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:14.737694 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 19:07:14.737821 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2559,"thread_start_us":79,"threads_started":1}
I20260929 19:07:14.739353 31265 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:07:14.739775 31155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca7fc4e721de4dd2996f0c5b6fad02ee: Bootstrap starting.
I20260929 19:07:14.740065 31265 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:07:14.740551 31155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca7fc4e721de4dd2996f0c5b6fad02ee: Neither blocks nor log segments found. Creating new log.
I20260929 19:07:14.740845 31155 log.cc:824] T 00000000000000000000000000000000 P ca7fc4e721de4dd2996f0c5b6fad02ee: Log is configured to *not* fsync() on all Append() calls
I20260929 19:07:14.742511 31159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:14.713523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50796 (local address 127.8.158.125:34223)
0929 19:07:14.713575 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:14.713577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:14.716579 (+ 3002us) server_negotiation.cc:407] Connection header received
0929 19:07:14.716667 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:14.716668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:14.716678 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:14.716706 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:14.718382 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.719099 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.722590 (+ 3491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.722913 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.722962 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:14.738983 (+ 16021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:14.738984 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:14.738984 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:07:14.738991 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:14.740787 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:14.741462 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:14.741463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:14.741464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:14.741531 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:14.741821 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:14.741822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:14.741822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:14.741960 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:14.742032 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:14.742354 (+ 322us) server_negotiation.cc:299] Negotiation successful
0929 19:07:14.742425 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 19:07:14.746901 31155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca7fc4e721de4dd2996f0c5b6fad02ee: No bootstrap required, opened a new log
I20260929 19:07:14.748858 31155 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca7fc4e721de4dd2996f0c5b6fad02ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3994f4bfba74c5b97d5fb326017c61b" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35661 } } peers { permanent_uuid: "ca7fc4e721de4dd2996f0c5b6fad02ee" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 34223 } } peers { permanent_uuid: "8a19f00263154bdfaf93f0f2dd4f60bb" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 37703 } }
I20260929 19:07:14.749002 31155 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca7fc4e721de4dd2996f0c5b6fad02ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca7fc4e721de4dd2996f0c5b6fad02ee, State: Initialized, Role: FOLLOWER
I20260929 19:07:14.749096 31155 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca7fc4e721de4dd2996f0c5b6fad02ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3994f4bfba74c5b97d5fb326017c61b" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35661 } } peers { permanent_uuid: "ca7fc4e721de4dd2996f0c5b6fad02ee" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 34223 } } peers { permanent_uuid: "8a19f00263154bdfaf93f0f2dd4f60bb" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 37703 } }
I20260929 19:07:14.749396 31155 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca7fc4e721de4dd2996f0c5b6fad02ee [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:07:14.751523 31128 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca7fc4e721de4dd2996f0c5b6fad02ee [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:07:14.751646 31129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a3994f4bfba74c5b97d5fb326017c61b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ca7fc4e721de4dd2996f0c5b6fad02ee"
I20260929 19:07:14.752691 31128 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ca7fc4e721de4dd2996f0c5b6fad02ee [term 1 FOLLOWER]: Refusing update from remote peer a3994f4bfba74c5b97d5fb326017c61b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:07:14.752997 31263 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a3994f4bfba74c5b97d5fb326017c61b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca7fc4e721de4dd2996f0c5b6fad02ee" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 34223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:07:14.753660 31129 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ca7fc4e721de4dd2996f0c5b6fad02ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3994f4bfba74c5b97d5fb326017c61b in term 1.
I20260929 19:07:14.755527 31295 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca7fc4e721de4dd2996f0c5b6fad02ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3994f4bfba74c5b97d5fb326017c61b. Latest consensus state: current_term: 1 leader_uuid: "a3994f4bfba74c5b97d5fb326017c61b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3994f4bfba74c5b97d5fb326017c61b" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35661 } } peers { permanent_uuid: "ca7fc4e721de4dd2996f0c5b6fad02ee" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 34223 } } peers { permanent_uuid: "8a19f00263154bdfaf93f0f2dd4f60bb" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 37703 } } }
I20260929 19:07:14.755820 31250 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3994f4bfba74c5b97d5fb326017c61b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a3994f4bfba74c5b97d5fb326017c61b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3994f4bfba74c5b97d5fb326017c61b" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35661 } } peers { permanent_uuid: "ca7fc4e721de4dd2996f0c5b6fad02ee" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 34223 } } peers { permanent_uuid: "8a19f00263154bdfaf93f0f2dd4f60bb" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 37703 } } }
I20260929 19:07:14.755863 31295 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca7fc4e721de4dd2996f0c5b6fad02ee [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:07:14.755898 31250 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3994f4bfba74c5b97d5fb326017c61b [sys.catalog]: This master's current role is: LEADER
I20260929 19:07:14.755929 31295 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca7fc4e721de4dd2996f0c5b6fad02ee [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a3994f4bfba74c5b97d5fb326017c61b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3994f4bfba74c5b97d5fb326017c61b" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35661 } } peers { permanent_uuid: "ca7fc4e721de4dd2996f0c5b6fad02ee" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 34223 } } peers { permanent_uuid: "8a19f00263154bdfaf93f0f2dd4f60bb" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 37703 } } }
I20260929 19:07:14.756223 31295 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca7fc4e721de4dd2996f0c5b6fad02ee [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:07:14.756892 31292 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca7fc4e721de4dd2996f0c5b6fad02ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3994f4bfba74c5b97d5fb326017c61b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3994f4bfba74c5b97d5fb326017c61b" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 35661 } } peers { permanent_uuid: "ca7fc4e721de4dd2996f0c5b6fad02ee" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 34223 } } peers { permanent_uuid: "8a19f00263154bdfaf93f0f2dd4f60bb" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 37703 } } }
I20260929 19:07:14.756968 31292 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca7fc4e721de4dd2996f0c5b6fad02ee [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:07:14.757074 31265 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3994f4bfba74c5b97d5fb326017c61b: Generated new TSK 0
I20260929 19:07:14.757462 31265 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:07:14.758526 31291 catalog_manager.cc:1259] Loaded cluster ID: 4ae563ed44ef4cc597a8413ab3570bb0
I20260929 19:07:14.758617 31291 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ca7fc4e721de4dd2996f0c5b6fad02ee: loading cluster ID for follower catalog manager: success
I20260929 19:07:14.759775 31291 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ca7fc4e721de4dd2996f0c5b6fad02ee: acquiring CA information for follower catalog manager: success
I20260929 19:07:14.760084 31291 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ca7fc4e721de4dd2996f0c5b6fad02ee: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 19:07:14.780913 31234 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:07:14.779465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33124 (local address 127.8.158.124:37703)
0929 19:07:14.779504 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:14.779506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:14.780333 (+ 827us) server_negotiation.cc:407] Connection header received
0929 19:07:14.780430 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:14.780431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:14.780442 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:14.780471 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:14.780853 (+ 382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33124: BlockingRecv error: recv got EOF from 127.0.0.1:33124 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260929 19:07:14.781814 31083 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:07:14.779792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33714 (local address 127.8.158.126:35661)
0929 19:07:14.779827 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:14.779828 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:14.781302 (+ 1474us) server_negotiation.cc:407] Connection header received
0929 19:07:14.781395 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:14.781396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:14.781405 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:14.781449 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:14.781761 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33714: BlockingRecv error: recv got EOF from 127.0.0.1:33714 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260929 19:07:14.782841 31159 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:07:14.780073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50806 (local address 127.8.158.125:34223)
0929 19:07:14.780116 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:14.780118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:14.782152 (+ 2034us) server_negotiation.cc:407] Connection header received
0929 19:07:14.782395 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:14.782396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:14.782406 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:14.782435 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:14.782790 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50806: BlockingRecv error: recv got EOF from 127.0.0.1:50806 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 19:07:14.787775 31159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:14.784351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50812 (local address 127.8.158.125:34223)
0929 19:07:14.784391 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:14.784393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:14.784593 (+ 200us) server_negotiation.cc:407] Connection header received
0929 19:07:14.784692 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:14.784693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:14.784722 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:14.784782 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:07:14.784981 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.785732 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.786872 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.787177 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.787217 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:14.787428 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:07:14.787537 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:07:14.787584 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:14.787693 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 19:07:14.787710 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
W20260929 19:07:14.790822 31234 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:07:14.790057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33136 (local address 127.8.158.124:37703)
0929 19:07:14.790092 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:14.790094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:14.790293 (+ 199us) server_negotiation.cc:407] Connection header received
0929 19:07:14.790443 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:14.790443 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:14.790458 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:14.790487 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:14.790774 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33136: BlockingRecv error: recv got EOF from 127.0.0.1:33136 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 19:07:14.796167 31159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:14.793048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50828 (local address 127.8.158.125:34223)
0929 19:07:14.793078 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:14.793080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:14.793316 (+ 236us) server_negotiation.cc:407] Connection header received
0929 19:07:14.793403 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:14.793404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:14.793412 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:14.793440 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:07:14.793595 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.794301 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.795396 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.795683 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.795720 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:14.795930 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:07:14.795960 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:07:14.795988 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:14.796089 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 19:07:14.796106 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260929 19:07:14.799072 31234 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:07:14.798362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33138 (local address 127.8.158.124:37703)
0929 19:07:14.798394 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:14.798396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:14.798608 (+ 212us) server_negotiation.cc:407] Connection header received
0929 19:07:14.798697 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:14.798698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:14.798707 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:14.798735 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:14.799029 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33138: BlockingRecv error: recv got EOF from 127.0.0.1:33138 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 19:07:14.805285 31159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:14.802151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50832 (local address 127.8.158.125:34223)
0929 19:07:14.802181 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:14.802183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:14.802452 (+ 269us) server_negotiation.cc:407] Connection header received
0929 19:07:14.802539 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:14.802540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:14.802547 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:14.802574 (+ 27us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:07:14.802715 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.803395 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.804409 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.804708 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.804761 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:14.805013 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:07:14.805042 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:07:14.805068 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:14.805158 (+ 90us) server_negotiation.cc:299] Negotiation successful
0929 19:07:14.805175 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
W20260929 19:07:14.808254 31234 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:07:14.807530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33150 (local address 127.8.158.124:37703)
0929 19:07:14.807564 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:14.807565 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:14.807783 (+ 218us) server_negotiation.cc:407] Connection header received
0929 19:07:14.807874 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:14.807875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:14.807883 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:14.807911 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:14.808208 (+ 297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33150: BlockingRecv error: recv got EOF from 127.0.0.1:33150 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 19:07:14.816403 31159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:14.813358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50848 (local address 127.8.158.125:34223)
0929 19:07:14.813389 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:14.813391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:14.813593 (+ 202us) server_negotiation.cc:407] Connection header received
0929 19:07:14.813681 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:14.813682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:14.813690 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:14.813716 (+ 26us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:07:14.813868 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.814586 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.815637 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.815899 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.815968 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:14.816176 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:07:14.816206 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:07:14.816234 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:14.816331 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 19:07:14.816347 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260929 19:07:14.819388 31234 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:07:14.818628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33160 (local address 127.8.158.124:37703)
0929 19:07:14.818673 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:14.818675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:14.818895 (+ 220us) server_negotiation.cc:407] Connection header received
0929 19:07:14.819005 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:14.819006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:14.819015 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:14.819044 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:14.819344 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33160: BlockingRecv error: recv got EOF from 127.0.0.1:33160 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
I20260929 19:07:14.831744 31159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:14.828577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50860 (local address 127.8.158.125:34223)
0929 19:07:14.828613 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:14.828615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:14.828843 (+ 228us) server_negotiation.cc:407] Connection header received
0929 19:07:14.828958 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:14.828959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:14.828969 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:14.828997 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:07:14.829164 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.829832 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.830974 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.831247 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.831291 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:14.831494 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:07:14.831526 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:07:14.831554 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:14.831661 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 19:07:14.831679 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260929 19:07:14.834784 31234 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:07:14.833983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33162 (local address 127.8.158.124:37703)
0929 19:07:14.834020 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:14.834022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:14.834254 (+ 232us) server_negotiation.cc:407] Connection header received
0929 19:07:14.834373 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:14.834373 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:14.834384 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:14.834414 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:14.834736 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33162: BlockingRecv error: recv got EOF from 127.0.0.1:33162 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 19:07:14.855707 31159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:14.852385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50876 (local address 127.8.158.125:34223)
0929 19:07:14.852424 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:14.852426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:14.852683 (+ 257us) server_negotiation.cc:407] Connection header received
0929 19:07:14.852782 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:14.852783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:14.852793 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:14.852824 (+ 31us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:07:14.853031 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.853720 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.854930 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.855204 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.855248 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:14.855450 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:07:14.855482 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:07:14.855511 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:14.855614 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 19:07:14.855633 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260929 19:07:14.858830 31234 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:07:14.858006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33176 (local address 127.8.158.124:37703)
0929 19:07:14.858048 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:14.858049 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:14.858275 (+ 226us) server_negotiation.cc:407] Connection header received
0929 19:07:14.858409 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:14.858411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:14.858423 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:14.858453 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:14.858785 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33176: BlockingRecv error: recv got EOF from 127.0.0.1:33176 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 19:07:14.894752 31159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:14.892199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50884 (local address 127.8.158.125:34223)
0929 19:07:14.892251 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:14.892253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:14.892349 (+ 96us) server_negotiation.cc:407] Connection header received
0929 19:07:14.892412 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:14.892413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:14.892424 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:14.892482 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:07:14.892597 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.893313 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.894015 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.894305 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.894402 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:14.894522 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:07:14.894556 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:07:14.894616 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:14.894660 (+ 44us) server_negotiation.cc:299] Negotiation successful
0929 19:07:14.894678 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260929 19:07:14.897020 31234 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:07:14.896481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33180 (local address 127.8.158.124:37703)
0929 19:07:14.896536 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:14.896537 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:14.896556 (+ 19us) server_negotiation.cc:407] Connection header received
0929 19:07:14.896611 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:14.896612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:14.896623 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:14.896692 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:14.896967 (+ 275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33180: BlockingRecv error: recv got EOF from 127.0.0.1:33180 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 19:07:14.964727 31159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:14.962092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50896 (local address 127.8.158.125:34223)
0929 19:07:14.962169 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:14.962172 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:14.962300 (+ 128us) server_negotiation.cc:407] Connection header received
0929 19:07:14.962382 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:14.962383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:14.962393 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:14.962461 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:07:14.962581 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.963266 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.964028 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:14.964210 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:14.964276 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:14.964445 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:07:14.964477 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:07:14.964535 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:14.964640 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 19:07:14.964665 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260929 19:07:14.967026 31234 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:07:14.966436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33186 (local address 127.8.158.124:37703)
0929 19:07:14.966512 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:14.966514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:14.966600 (+ 86us) server_negotiation.cc:407] Connection header received
0929 19:07:14.966643 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:14.966643 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:14.966653 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:14.966700 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:14.966971 (+ 271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33186: BlockingRecv error: recv got EOF from 127.0.0.1:33186 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 19:07:15.098392 31159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:15.096143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50900 (local address 127.8.158.125:34223)
0929 19:07:15.096213 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:15.096215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:15.096310 (+ 95us) server_negotiation.cc:407] Connection header received
0929 19:07:15.096377 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:15.096378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:15.096387 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:15.096432 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:07:15.096600 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:15.097068 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:15.097843 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:15.098015 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:15.098062 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:15.098183 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:07:15.098207 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:07:15.098244 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:15.098278 (+ 34us) server_negotiation.cc:299] Negotiation successful
0929 19:07:15.098291 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260929 19:07:15.100636 31234 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:07:15.100166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33202 (local address 127.8.158.124:37703)
0929 19:07:15.100221 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:15.100223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:15.100231 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:07:15.100296 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:15.100297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:15.100305 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:15.100345 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:15.100592 (+ 247us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33202: BlockingRecv error: recv got EOF from 127.0.0.1:33202 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 19:07:15.360107 31159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:15.357859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50912 (local address 127.8.158.125:34223)
0929 19:07:15.357935 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:15.357937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:15.357962 (+ 25us) server_negotiation.cc:407] Connection header received
0929 19:07:15.358029 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:15.358030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:15.358038 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:15.358102 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:07:15.358211 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:15.358751 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:15.359509 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:15.359691 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:15.359743 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:15.359863 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:07:15.359887 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:07:15.359925 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:15.359998 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 19:07:15.360013 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 19:07:15.362280 31234 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:07:15.361720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33214 (local address 127.8.158.124:37703)
0929 19:07:15.361780 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:15.361782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:15.361826 (+ 44us) server_negotiation.cc:407] Connection header received
0929 19:07:15.361891 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:15.361891 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:15.361901 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:15.361940 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:15.362218 (+ 278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33214: BlockingRecv error: recv got EOF from 127.0.0.1:33214 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 19:07:15.676091 31247 catalog_manager.cc:1259] Loaded cluster ID: 4ae563ed44ef4cc597a8413ab3570bb0
I20260929 19:07:15.676153 31247 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8a19f00263154bdfaf93f0f2dd4f60bb: loading cluster ID for follower catalog manager: success
I20260929 19:07:15.676930 31247 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8a19f00263154bdfaf93f0f2dd4f60bb: acquiring CA information for follower catalog manager: success
I20260929 19:07:15.677182 31247 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8a19f00263154bdfaf93f0f2dd4f60bb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 19:07:15.877579 31413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:15.875270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50924 (local address 127.8.158.125:34223)
0929 19:07:15.875383 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:15.875385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:15.875410 (+ 25us) server_negotiation.cc:407] Connection header received
0929 19:07:15.875462 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:15.875462 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:15.875472 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:15.875525 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:07:15.875682 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:15.876186 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:15.876933 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:15.877120 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:15.877202 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:15.877323 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:07:15.877351 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:07:15.877392 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:15.877435 (+ 43us) server_negotiation.cc:299] Negotiation successful
0929 19:07:15.877458 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":70,"threads_started":1}
I20260929 19:07:15.881769 31419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:15.879318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33226 (local address 127.8.158.124:37703)
0929 19:07:15.879434 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:15.879436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:15.879445 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:07:15.879493 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:15.879493 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:15.879563 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:15.879637 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:07:15.879756 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:15.880266 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:15.880985 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:15.881207 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:15.881287 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:15.881450 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:07:15.881562 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:07:15.881646 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:15.881669 (+ 23us) server_negotiation.cc:299] Negotiation successful
0929 19:07:15.881688 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":54,"threads_started":1}
I20260929 19:07:15.886125 31419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:15.883453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33230 (local address 127.8.158.124:37703)
0929 19:07:15.883492 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:15.883494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:15.883621 (+ 127us) server_negotiation.cc:407] Connection header received
0929 19:07:15.883669 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:15.883670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:15.883679 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:15.883797 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:07:15.883938 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:15.884459 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:15.885368 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:15.885648 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:15.885685 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:15.885877 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:07:15.885904 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:07:15.885972 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:15.886024 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 19:07:15.886041 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 19:07:15.886612 31413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:15.883515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50930 (local address 127.8.158.125:34223)
0929 19:07:15.883555 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:15.883556 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:15.883844 (+ 288us) server_negotiation.cc:407] Connection header received
0929 19:07:15.883935 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:15.883936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:15.883948 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:15.883996 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:07:15.884099 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:15.884873 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:15.885699 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:15.886008 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:15.886072 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:15.886294 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:07:15.886359 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:07:15.886394 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:15.886523 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 19:07:15.886548 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 19:07:15.886825 31426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:15.883354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33722 (local address 127.8.158.126:35661)
0929 19:07:15.883544 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:15.883546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:15.883554 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:07:15.883597 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:15.883598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:15.883650 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:15.883725 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:07:15.883889 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:15.884766 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:15.885760 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:15.886105 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:15.886167 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:15.886435 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:07:15.886606 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:07:15.886674 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:15.886729 (+ 55us) server_negotiation.cc:299] Negotiation successful
0929 19:07:15.886756 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":57,"threads_started":1}
I20260929 19:07:15.888429 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 31012
I20260929 19:07:15.893831 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 31087
I20260929 19:07:15.898564 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 31162
2026-09-29T19:07:15Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2801 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:07:15 dist-test-slave-nmsz krb5kdc[31430](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:07:15 dist-test-slave-nmsz krb5kdc[31430](info): set up 2 sockets
Sep 29 19:07:15 dist-test-slave-nmsz krb5kdc[31430](info): commencing operation
krb5kdc: starting...
W20260929 19:07:16.934616 8825 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/krb5kdc/test-user.keytab.
Sep 29 19:07:17 dist-test-slave-nmsz krb5kdc[31430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:07:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:07:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.158.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
Entry for principal kudu/127.8.158.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab.
I20260929 19:07:17.045048 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:37381
--webserver_interface=127.8.158.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:35773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:37381,127.8.158.125:38439,127.8.158.124:34097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:07:17.121660 31443 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 19:07:17.121845 31443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:07:17.121861 31443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:07:17.121888 31443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:07:17.122963 31443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:07:17.122999 31443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:07:17.123013 31443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:07:17.123025 31443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:07:17.123037 31443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:07:17.124394 31443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:35773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:37381,127.8.158.125:38439,127.8.158.124:34097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.126.keytab
--principal=kudu/127.8.158.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.126:37381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.158.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:07:17.124564 31443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:07:17.124709 31443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:07:17.129148 31449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:07:17.129262 31448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:07:17.129307 31443 server_base.cc:1034] running on GCE node
W20260929 19:07:17.129176 31451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:07:17.129546 31443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:07:17.129822 31443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:07:17.130996 31443 hybrid_clock.cc:648] HybridClock initialized: now 1790708837130976 us; error 37 us; skew 500 ppm
Sep 29 19:07:17 dist-test-slave-nmsz krb5kdc[31430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708837, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:07:17.133335 31443 init.cc:377] Logged in from keytab as kudu/127.8.158.126@KRBTEST.COM (short username kudu)
I20260929 19:07:17.133716 31443 webserver.cc:460] Webserver started at http://127.8.158.126:39121/ using document root <none> and password file <none>
I20260929 19:07:17.133924 31443 fs_manager.cc:359] Metadata directory not provided
I20260929 19:07:17.133989 31443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:07:17.134131 31443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:07:17.135128 31443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-0/data/instance:
uuid: "3e45422dfed440bc9bff6bedc922a306"
format_stamp: "Formatted at 2026-09-29 19:07:17 on dist-test-slave-nmsz"
server_key: "1b925dd24a643747df0bde90e6367157"
server_key_iv: "3a2c75b34891a8dac46cd6b949a14d7d"
server_key_version: "encryptionkey@0"
I20260929 19:07:17.135475 31443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-0/wal/instance:
uuid: "3e45422dfed440bc9bff6bedc922a306"
format_stamp: "Formatted at 2026-09-29 19:07:17 on dist-test-slave-nmsz"
server_key: "1b925dd24a643747df0bde90e6367157"
server_key_iv: "3a2c75b34891a8dac46cd6b949a14d7d"
server_key_version: "encryptionkey@0"
I20260929 19:07:17.136951 31443 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:07:17.137595 31458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:07:17.137744 31443 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:07:17.137868 31443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-0/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-0/wal
uuid: "3e45422dfed440bc9bff6bedc922a306"
format_stamp: "Formatted at 2026-09-29 19:07:17 on dist-test-slave-nmsz"
server_key: "1b925dd24a643747df0bde90e6367157"
server_key_iv: "3a2c75b34891a8dac46cd6b949a14d7d"
server_key_version: "encryptionkey@0"
I20260929 19:07:17.137948 31443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:07:17.146287 31443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:07:17.148424 31443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:07:17.148568 31443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:07:17.152271 31443 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.126:37381
I20260929 19:07:17.152300 31510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.126:37381 every 8 connection(s)
I20260929 19:07:17.152598 31443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-0/data/info.pb
I20260929 19:07:17.153529 31511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:07:17.154984 31511 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37381 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:07:17.159009 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 31443
I20260929 19:07:17.159117 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-0/wal/instance
I20260929 19:07:17.159292 8825 external_mini_cluster.cc:1428] Setting key 31b877f8604e1d6df521f4bacc1c5b7d
Sep 29 19:07:17 dist-test-slave-nmsz krb5kdc[31430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708837, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
I20260929 19:07:17.165128 31514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:17.155593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37528 (local address 127.8.158.126:37381)
0929 19:07:17.156052 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:17.156056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:17.156090 (+ 34us) server_negotiation.cc:407] Connection header received
0929 19:07:17.156455 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:17.156457 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:17.156472 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:17.156562 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:17.157047 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:17.157826 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:17.158723 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:17.159023 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:17.159146 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:17.161407 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:17.161415 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:17.161415 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:07:17.161424 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:17.163368 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:17.163731 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:17.163734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:17.163735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:17.163838 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:17.164046 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:17.164046 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:17.164047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:17.164332 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:17.164429 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:17.164634 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 19:07:17.164784 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":429,"thread_start_us":71,"threads_started":1}
I20260929 19:07:17.165133 31513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:17.155570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:37381 (local address 127.0.0.1:37528)
0929 19:07:17.155735 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 19:07:17.155765 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 19:07:17.156291 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:07:17.156566 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:07:17.156568 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:07:17.156818 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:07:17.156954 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:07:17.156958 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:07:17.157938 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:07:17.157939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:07:17.158635 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:07:17.158637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:07:17.159131 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:07:17.159132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:07:17.159193 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:17.159664 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:07:17.159671 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:07:17.161270 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:07:17.163458 (+ 2188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:07:17.163464 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:07:17.163466 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:07:17.163644 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:07:17.163912 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:07:17.163913 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:07:17.163913 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:07:17.163986 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:07:17.164403 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:07:17.164407 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:07:17.164503 (+ 96us) client_negotiation.cc:769] Sending connection context
0929 19:07:17.164598 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 19:07:17.164884 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":120,"thread_start_us":79,"threads_started":1}
I20260929 19:07:17.166357 31511 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.158.125" port: 38439 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:07:17.166760 31513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:17.166591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.125:38439 (local address 127.0.0.1:38094)
0929 19:07:17.166627 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0929 19:07:17.166702 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.125:38439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260929 19:07:17.166937 31461 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.158.125:38439: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.8.158.125@KRBTEST.COM; defaulting to no policy
I20260929 19:07:17.167778 31513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:17.167540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.125:38439 (local address 127.0.0.1:38106)
0929 19:07:17.167684 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 19:07:17.167714 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.125:38439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260929 19:07:17.167920 31511 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.158.125:38439: Network error: Client connection negotiation failed: client connection to 127.8.158.125:38439: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.125.keytab.
Entry for principal kudu/127.8.158.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.125.keytab.
I20260929 19:07:17.184320 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.125:38439
--webserver_interface=127.8.158.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.125.keytab
--principal=kudu/127.8.158.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:35773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:37381,127.8.158.125:38439,127.8.158.124:34097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:07:17.217131 31511 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.158.125" port: 38439 } attempt: 1
I20260929 19:07:17.217624 31513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:17.217477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.125:38439 (local address 127.0.0.1:38108)
0929 19:07:17.217536 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0929 19:07:17.217582 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.125:38439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 19:07:17.218065 31513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:17.217906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.125:38439 (local address 127.0.0.1:38112)
0929 19:07:17.217999 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0929 19:07:17.218033 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.125:38439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260929 19:07:17.218232 31511 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.158.125:38439: Network error: Client connection negotiation failed: client connection to 127.8.158.125:38439: connect: Connection refused (error 111)
W20260929 19:07:17.261191 31518 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 19:07:17.261371 31518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:07:17.261387 31518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:07:17.261439 31518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:07:17.262542 31518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:07:17.262575 31518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:07:17.262589 31518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:07:17.262634 31518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:07:17.262661 31518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:07:17.264079 31518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:35773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:37381,127.8.158.125:38439,127.8.158.124:34097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.125.keytab
--principal=kudu/127.8.158.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.125:38439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.158.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:07:17.264310 31518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:07:17.264513 31518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:07:17.268908 31526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:07:17.268982 31524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:07:17.269104 31523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:07:17.269222 31518 server_base.cc:1034] running on GCE node
I20260929 19:07:17.269428 31518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:07:17.269647 31518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:07:17.270828 31518 hybrid_clock.cc:648] HybridClock initialized: now 1790708837270784 us; error 27 us; skew 500 ppm
Sep 29 19:07:17 dist-test-slave-nmsz krb5kdc[31430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708837, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:07:17.273130 31518 init.cc:377] Logged in from keytab as kudu/127.8.158.125@KRBTEST.COM (short username kudu)
I20260929 19:07:17.273476 31518 webserver.cc:460] Webserver started at http://127.8.158.125:46155/ using document root <none> and password file <none>
I20260929 19:07:17.273636 31518 fs_manager.cc:359] Metadata directory not provided
I20260929 19:07:17.273674 31518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:07:17.273754 31518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:07:17.274742 31518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-1/data/instance:
uuid: "5078132eba2d40659afc2b299bad79ff"
format_stamp: "Formatted at 2026-09-29 19:07:17 on dist-test-slave-nmsz"
server_key: "1ec3dcdc5547ad0974a023a98b7cff58"
server_key_iv: "749b8447ec61dd86e49e61d5bf334002"
server_key_version: "encryptionkey@0"
I20260929 19:07:17.275030 31518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-1/wal/instance:
uuid: "5078132eba2d40659afc2b299bad79ff"
format_stamp: "Formatted at 2026-09-29 19:07:17 on dist-test-slave-nmsz"
server_key: "1ec3dcdc5547ad0974a023a98b7cff58"
server_key_iv: "749b8447ec61dd86e49e61d5bf334002"
server_key_version: "encryptionkey@0"
I20260929 19:07:17.276283 31518 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:07:17.276942 31533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:07:17.277084 31518 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:07:17.277158 31518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-1/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-1/wal
uuid: "5078132eba2d40659afc2b299bad79ff"
format_stamp: "Formatted at 2026-09-29 19:07:17 on dist-test-slave-nmsz"
server_key: "1ec3dcdc5547ad0974a023a98b7cff58"
server_key_iv: "749b8447ec61dd86e49e61d5bf334002"
server_key_version: "encryptionkey@0"
I20260929 19:07:17.277207 31518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:07:17.286412 31511 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.158.125" port: 38439 } attempt: 2
I20260929 19:07:17.286787 31513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:17.286655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.125:38439 (local address 127.0.0.1:38120)
0929 19:07:17.286711 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0929 19:07:17.286750 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.125:38439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 19:07:17.287117 31513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:17.287040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.125:38439 (local address 127.0.0.1:38130)
0929 19:07:17.287077 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0929 19:07:17.287100 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.125:38439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260929 19:07:17.287259 31511 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.158.125:38439: Network error: Client connection negotiation failed: client connection to 127.8.158.125:38439: connect: Connection refused (error 111)
I20260929 19:07:17.299943 31518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:07:17.302054 31518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:07:17.302204 31518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:07:17.305816 31518 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.125:38439
I20260929 19:07:17.305874 31585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.125:38439 every 8 connection(s)
I20260929 19:07:17.306205 31518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-1/data/info.pb
I20260929 19:07:17.307170 31586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:07:17.308281 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 31518
I20260929 19:07:17.308364 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-1/wal/instance
I20260929 19:07:17.308557 8825 external_mini_cluster.cc:1428] Setting key 34e9f6f67f6d87235e8a0983a156d572
I20260929 19:07:17.308722 31586 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37381 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 19:07:17 dist-test-slave-nmsz krb5kdc[31430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708837, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.125@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.158.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.158.124@KRBTEST.COM" created.
I20260929 19:07:17.318531 31514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:17.309415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37542 (local address 127.8.158.126:37381)
0929 19:07:17.309478 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:17.309480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:17.309700 (+ 220us) server_negotiation.cc:407] Connection header received
0929 19:07:17.310233 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:17.310234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:17.310247 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:17.310343 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:17.311039 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:17.311505 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:17.313100 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:17.313295 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:17.313360 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:17.315720 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:17.315721 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:17.315722 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:07:17.315729 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:17.317136 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:17.317529 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:17.317531 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:17.317531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:17.317612 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:17.317869 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:17.317870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:17.317870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:17.317986 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:17.318062 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:17.318358 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 19:07:17.318440 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 19:07:17.318764 31588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:17.309396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:37381 (local address 127.0.0.1:37542)
0929 19:07:17.309558 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 19:07:17.309604 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 19:07:17.310142 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:07:17.310424 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:07:17.310427 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:07:17.310678 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:07:17.310947 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:07:17.310953 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:07:17.311598 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:07:17.311599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:07:17.312997 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:07:17.313001 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:07:17.313395 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:07:17.313396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:07:17.313565 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:17.314065 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:07:17.314075 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:07:17.315627 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:07:17.317225 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:07:17.317232 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:07:17.317234 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:07:17.317453 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:07:17.317678 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:07:17.317679 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:07:17.317679 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:07:17.317796 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:07:17.318052 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:07:17.318059 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:07:17.318204 (+ 145us) client_negotiation.cc:769] Sending connection context
0929 19:07:17.318321 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 19:07:17.318490 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":82,"threads_started":1}
I20260929 19:07:17.319299 31586 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.158.125" port: 38439 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 19:07:17 dist-test-slave-nmsz krb5kdc[31430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708837, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.125@KRBTEST.COM for kudu/127.8.158.125@KRBTEST.COM
I20260929 19:07:17.326792 31588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:17.319552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.125:38439 (local address 127.0.0.1:38136)
0929 19:07:17.319749 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 19:07:17.319759 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:07:17.319805 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:07:17.319980 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:07:17.319981 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:07:17.320091 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:07:17.320130 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:07:17.320131 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:07:17.320946 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:07:17.320947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:07:17.321816 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:07:17.321819 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:07:17.322371 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:07:17.322372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:07:17.322451 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:17.322810 (+ 359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:07:17.322816 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:07:17.324081 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:07:17.325799 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:07:17.325800 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:07:17.325801 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:07:17.326025 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:07:17.326205 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:07:17.326206 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:07:17.326206 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:07:17.326256 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:07:17.326571 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:07:17.326572 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:07:17.326612 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 19:07:17.326665 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 19:07:17.326734 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183}
I20260929 19:07:17.326974 31591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:17.319619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38136 (local address 127.8.158.125:38439)
0929 19:07:17.319814 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:17.319818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:17.319838 (+ 20us) server_negotiation.cc:407] Connection header received
0929 19:07:17.319859 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:17.319861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:17.319872 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:17.319936 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:17.320210 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:17.320843 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:17.321894 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:17.322257 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:17.322352 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:17.324183 (+ 1831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:17.324190 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:17.324190 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:07:17.324196 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:17.325707 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:17.326095 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:17.326097 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:17.326097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:17.326147 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:17.326305 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:17.326305 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:17.326305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:17.326467 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:17.326592 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:17.326690 (+ 98us) server_negotiation.cc:299] Negotiation successful
0929 19:07:17.326792 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":169,"thread_start_us":80,"threads_started":1}
I20260929 19:07:17.327623 31586 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.158.124" port: 34097 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:07:17.327916 31588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:17.327810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.124:34097 (local address 127.0.0.1:59836)
0929 19:07:17.327852 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0929 19:07:17.327891 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.124:34097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260929 19:07:17.328030 31535 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.158.124:34097: connect: Connection refused (error 111)
I20260929 19:07:17.328394 31588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:17.328300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.124:34097 (local address 127.0.0.1:59846)
0929 19:07:17.328344 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0929 19:07:17.328367 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.124:34097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260929 19:07:17.328578 31586 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.158.124:34097: Network error: Client connection negotiation failed: client connection to 127.8.158.124:34097: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.158.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.124.keytab.
Entry for principal kudu/127.8.158.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.124.keytab.
I20260929 19:07:17.333559 8825 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkdiMiM/build/release/bin/kudu
/tmp/dist-test-taskkdiMiM/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.124:34097
--webserver_interface=127.8.158.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.124.keytab
--principal=kudu/127.8.158.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.158.84:35773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.158.126:37381,127.8.158.125:38439,127.8.158.124:34097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:07:17.377785 31586 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.158.124" port: 34097 } attempt: 1
I20260929 19:07:17.378202 31588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:17.378081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.124:34097 (local address 127.0.0.1:59862)
0929 19:07:17.378140 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0929 19:07:17.378174 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.124:34097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 19:07:17.378438 31511 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.158.125" port: 38439 } attempt: 3
I20260929 19:07:17.378587 31588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:17.378479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.124:34097 (local address 127.0.0.1:59870)
0929 19:07:17.378518 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0929 19:07:17.378542 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.124:34097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260929 19:07:17.378684 31586 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.158.124:34097: Network error: Client connection negotiation failed: client connection to 127.8.158.124:34097: connect: Connection refused (error 111)
Sep 29 19:07:17 dist-test-slave-nmsz krb5kdc[31430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708837, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for kudu/127.8.158.125@KRBTEST.COM
I20260929 19:07:17.385399 31591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:17.378778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38146 (local address 127.8.158.125:38439)
0929 19:07:17.378822 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:17.378824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:17.378834 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:07:17.378865 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:17.378866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:17.378876 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:17.378979 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:17.379285 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:17.379967 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:17.380656 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:17.381025 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:17.381143 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:17.382969 (+ 1826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:17.382970 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:17.382971 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:07:17.382977 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:17.384336 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:17.384665 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:17.384666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:17.384666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:17.384718 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:17.384915 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:17.384917 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:17.384917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:17.385032 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:17.385103 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:17.385243 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 19:07:17.385315 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260929 19:07:17.385404 31513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:17.378696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.125:38439 (local address 127.0.0.1:38146)
0929 19:07:17.378743 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0929 19:07:17.378752 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:07:17.378795 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:07:17.378983 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:07:17.378984 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:07:17.379141 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:07:17.379199 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:07:17.379201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:07:17.380040 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:07:17.380042 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:07:17.380576 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:07:17.380577 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:07:17.381145 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:07:17.381146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:07:17.381241 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:17.381699 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:07:17.381704 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:07:17.382876 (+ 1172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:07:17.384428 (+ 1552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:07:17.384429 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:07:17.384430 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:07:17.384596 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:07:17.384793 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:07:17.384794 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:07:17.384794 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:07:17.384855 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:07:17.385096 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:07:17.385097 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:07:17.385139 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 19:07:17.385220 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 19:07:17.385315 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 19:07:17.385735 31511 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.158.124" port: 34097 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:07:17.386062 31513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:17.385938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.124:34097 (local address 127.0.0.1:59874)
0929 19:07:17.385997 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0929 19:07:17.386033 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.124:34097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260929 19:07:17.386406 31513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:17.386296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.124:34097 (local address 127.0.0.1:59888)
0929 19:07:17.386355 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0929 19:07:17.386382 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.124:34097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 19:07:17.386523 31511 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.158.124:34097: Network error: Client connection negotiation failed: client connection to 127.8.158.124:34097: connect: Connection refused (error 111)
W20260929 19:07:17.412076 31593 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 19:07:17.412253 31593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:07:17.412271 31593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:07:17.412289 31593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:07:17.413344 31593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:07:17.413380 31593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:07:17.413394 31593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:07:17.413406 31593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:07:17.413419 31593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:07:17.414860 31593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.158.84:35773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.158.126:37381,127.8.158.125:38439,127.8.158.124:34097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/krb5kdc/kudu_127.8.158.124.keytab
--principal=kudu/127.8.158.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.158.124:34097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.158.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 16:56:54 UTC on ccycloud.quasar-kugsms.root.comops.site
build id 4680447
I20260929 19:07:17.415032 31593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:07:17.415179 31593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 19:07:17.417665 31511 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.158.124" port: 34097 } attempt: 1
I20260929 19:07:17.418011 31513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:17.417888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.124:34097 (local address 127.0.0.1:59896)
0929 19:07:17.417941 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0929 19:07:17.417981 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.124:34097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 19:07:17.418408 31513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:17.418289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.124:34097 (local address 127.0.0.1:59906)
0929 19:07:17.418353 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0929 19:07:17.418387 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.158.124:34097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 19:07:17.418529 31511 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.158.124:34097: Network error: Client connection negotiation failed: client connection to 127.8.158.124:34097: connect: Connection refused (error 111)
W20260929 19:07:17.419600 31599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:07:17.419606 31601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:07:17.419608 31598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:07:17.419929 31593 server_base.cc:1034] running on GCE node
I20260929 19:07:17.420145 31593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:07:17.420396 31593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:07:17.421551 31593 hybrid_clock.cc:648] HybridClock initialized: now 1790708837421535 us; error 27 us; skew 500 ppm
Sep 29 19:07:17 dist-test-slave-nmsz krb5kdc[31430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708837, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:07:17.423774 31593 init.cc:377] Logged in from keytab as kudu/127.8.158.124@KRBTEST.COM (short username kudu)
I20260929 19:07:17.424121 31593 webserver.cc:460] Webserver started at http://127.8.158.124:42817/ using document root <none> and password file <none>
I20260929 19:07:17.424286 31593 fs_manager.cc:359] Metadata directory not provided
I20260929 19:07:17.424328 31593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:07:17.424412 31593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:07:17.425309 31593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-2/data/instance:
uuid: "d2f827e6065a4104b2f63037514cafb1"
format_stamp: "Formatted at 2026-09-29 19:07:17 on dist-test-slave-nmsz"
server_key: "52410a4a53130cf7e7a4bc3d6c25ee7c"
server_key_iv: "3d8d7981be11b42857a36df1e75ecb84"
server_key_version: "encryptionkey@0"
I20260929 19:07:17.425591 31593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-2/wal/instance:
uuid: "d2f827e6065a4104b2f63037514cafb1"
format_stamp: "Formatted at 2026-09-29 19:07:17 on dist-test-slave-nmsz"
server_key: "52410a4a53130cf7e7a4bc3d6c25ee7c"
server_key_iv: "3d8d7981be11b42857a36df1e75ecb84"
server_key_version: "encryptionkey@0"
I20260929 19:07:17.426821 31593 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 19:07:17.427457 31608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:07:17.427619 31593 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:07:17.427696 31593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-2/data,/tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-2/wal
uuid: "d2f827e6065a4104b2f63037514cafb1"
format_stamp: "Formatted at 2026-09-29 19:07:17 on dist-test-slave-nmsz"
server_key: "52410a4a53130cf7e7a4bc3d6c25ee7c"
server_key_iv: "3d8d7981be11b42857a36df1e75ecb84"
server_key_version: "encryptionkey@0"
I20260929 19:07:17.427747 31593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:07:17.437059 31593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:07:17.438758 31593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:07:17.438869 31593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:07:17.442384 31593 rpc_server.cc:307] RPC server started. Bound to: 127.8.158.124:34097
I20260929 19:07:17.442404 31660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.158.124:34097 every 8 connection(s)
I20260929 19:07:17.442708 31593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-2/data/info.pb
I20260929 19:07:17.443594 31661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:07:17.444927 31661 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37381 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:07:17.446832 31586 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.158.124" port: 34097 } attempt: 2
I20260929 19:07:17.447413 8825 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkdiMiM/build/release/bin/kudu as pid 31593
I20260929 19:07:17.447539 8825 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkdiMiM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708573736807-8825-0/minicluster-data/master-2/wal/instance
I20260929 19:07:17.447749 8825 external_mini_cluster.cc:1428] Setting key 786b2060793926ddcd8e9617460fc456
Sep 29 19:07:17 dist-test-slave-nmsz krb5kdc[31430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708837, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.124@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
Sep 29 19:07:17 dist-test-slave-nmsz krb5kdc[31430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708837, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.125@KRBTEST.COM for kudu/127.8.158.124@KRBTEST.COM
Sep 29 19:07:17 dist-test-slave-nmsz krb5kdc[31430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.158.124@KRBTEST.COM
I20260929 19:07:17.454833 31514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:17.445519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37544 (local address 127.8.158.126:37381)
0929 19:07:17.445579 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:17.445581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:17.445783 (+ 202us) server_negotiation.cc:407] Connection header received
0929 19:07:17.446251 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:17.446252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:17.446265 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:17.446333 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:17.446907 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:17.447558 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:17.448795 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:17.449056 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:17.449124 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:17.451524 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:17.451525 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:17.451526 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:07:17.451533 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:17.453209 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:17.453639 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:17.453640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:17.453640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:17.453699 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:17.454137 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:17.454138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:17.454138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:17.454243 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:17.454324 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:17.454652 (+ 328us) server_negotiation.cc:299] Negotiation successful
0929 19:07:17.454729 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 19:07:17.454937 31663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:17.445523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.126:37381 (local address 127.0.0.1:37544)
0929 19:07:17.445678 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 19:07:17.445711 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 19:07:17.446148 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:07:17.446409 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:07:17.446413 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:07:17.446634 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:07:17.446802 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:07:17.446806 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:07:17.447646 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:07:17.447647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:07:17.448717 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:07:17.448719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:07:17.449161 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:07:17.449162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:07:17.449275 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:17.449755 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:07:17.449763 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:07:17.451449 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:07:17.453299 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:07:17.453305 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:07:17.453307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:07:17.453580 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:07:17.453763 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:07:17.453764 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:07:17.453764 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:07:17.453868 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:07:17.454336 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:07:17.454343 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:07:17.454501 (+ 158us) client_negotiation.cc:769] Sending connection context
0929 19:07:17.454613 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 19:07:17.454761 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":75,"threads_started":1}
I20260929 19:07:17.455459 31661 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.158.125" port: 38439 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:07:17.455986 31588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:17.447087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.124:34097 (local address 127.0.0.1:59914)
0929 19:07:17.447140 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0929 19:07:17.447150 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:07:17.447192 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:07:17.447521 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:07:17.447522 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:07:17.447692 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:07:17.447732 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:07:17.447733 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:07:17.448959 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:07:17.448960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:07:17.449757 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:07:17.449759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:07:17.450389 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:07:17.450390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:07:17.450451 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:17.450879 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:07:17.450883 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:07:17.452509 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:07:17.454760 (+ 2251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:07:17.454761 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:07:17.454762 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:07:17.454986 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:07:17.455247 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:07:17.455248 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:07:17.455249 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:07:17.455316 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:07:17.455634 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:07:17.455635 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:07:17.455697 (+ 62us) client_negotiation.cc:769] Sending connection context
0929 19:07:17.455767 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 19:07:17.455869 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 19:07:17.456440 31664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:17.447163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59914 (local address 127.8.158.124:34097)
0929 19:07:17.447331 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:17.447336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:17.447360 (+ 24us) server_negotiation.cc:407] Connection header received
0929 19:07:17.447397 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:17.447399 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:17.447419 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:17.447503 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:17.447806 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:17.448875 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:17.449835 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:17.450266 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:17.450389 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:17.452601 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:17.452608 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:17.452609 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:07:17.452615 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:17.454671 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:17.455089 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:17.455092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:17.455092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:17.455161 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:17.455387 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:17.455388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:17.455388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:17.455561 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:17.455635 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:17.456207 (+ 572us) server_negotiation.cc:299] Negotiation successful
0929 19:07:17.456346 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":90,"threads_started":1}
I20260929 19:07:17.457778 31666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:17.448054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59928 (local address 127.8.158.124:34097)
0929 19:07:17.448338 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:17.448339 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:17.448346 (+ 7us) server_negotiation.cc:407] Connection header received
0929 19:07:17.448362 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:17.448363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:17.448371 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:17.448433 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:17.449189 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:17.449867 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:17.450750 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:17.451057 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:17.451126 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:17.453715 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:17.453716 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:17.453716 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:07:17.453722 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:17.456571 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:17.456991 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:17.456992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:17.456992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:17.457047 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:17.457248 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:17.457249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:17.457249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:17.457377 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:17.457504 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:17.457591 (+ 87us) server_negotiation.cc:299] Negotiation successful
0929 19:07:17.457659 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":80,"threads_started":1}
I20260929 19:07:17.458120 31586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5078132eba2d40659afc2b299bad79ff: Bootstrap starting.
I20260929 19:07:17.459115 31586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5078132eba2d40659afc2b299bad79ff: Neither blocks nor log segments found. Creating new log.
I20260929 19:07:17.459444 31586 log.cc:824] T 00000000000000000000000000000000 P 5078132eba2d40659afc2b299bad79ff: Log is configured to *not* fsync() on all Append() calls
Sep 29 19:07:17 dist-test-slave-nmsz krb5kdc[31430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708837, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.124@KRBTEST.COM for kudu/127.8.158.125@KRBTEST.COM
I20260929 19:07:17.460170 31586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5078132eba2d40659afc2b299bad79ff: No bootstrap required, opened a new log
I20260929 19:07:17.461813 31586 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5078132eba2d40659afc2b299bad79ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e45422dfed440bc9bff6bedc922a306" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37381 } } peers { permanent_uuid: "5078132eba2d40659afc2b299bad79ff" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 38439 } } peers { permanent_uuid: "d2f827e6065a4104b2f63037514cafb1" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 34097 } }
I20260929 19:07:17.461944 31586 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5078132eba2d40659afc2b299bad79ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5078132eba2d40659afc2b299bad79ff, State: Initialized, Role: FOLLOWER
I20260929 19:07:17.462033 31586 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5078132eba2d40659afc2b299bad79ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e45422dfed440bc9bff6bedc922a306" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37381 } } peers { permanent_uuid: "5078132eba2d40659afc2b299bad79ff" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 38439 } } peers { permanent_uuid: "d2f827e6065a4104b2f63037514cafb1" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 34097 } }
I20260929 19:07:17.462270 31668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5078132eba2d40659afc2b299bad79ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e45422dfed440bc9bff6bedc922a306" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37381 } } peers { permanent_uuid: "5078132eba2d40659afc2b299bad79ff" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 38439 } } peers { permanent_uuid: "d2f827e6065a4104b2f63037514cafb1" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 34097 } } }
I20260929 19:07:17.462359 31586 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5078132eba2d40659afc2b299bad79ff [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:07:17.462589 31663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:17.455692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.125:38439 (local address 127.0.0.1:38150)
0929 19:07:17.455848 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 19:07:17.455858 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:07:17.455920 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:07:17.456040 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:07:17.456041 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:07:17.456228 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:07:17.456262 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:07:17.456264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:07:17.457064 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:07:17.457065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:07:17.457771 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:07:17.457774 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:07:17.458437 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:07:17.458438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:07:17.458502 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:17.458896 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:07:17.458901 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:07:17.460147 (+ 1246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:07:17.461550 (+ 1403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:07:17.461550 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:07:17.461551 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:07:17.461710 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:07:17.461906 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:07:17.461907 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:07:17.461907 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:07:17.461960 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:07:17.462281 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:07:17.462282 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:07:17.462349 (+ 67us) client_negotiation.cc:769] Sending connection context
0929 19:07:17.462424 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 19:07:17.462504 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142}
I20260929 19:07:17.462602 31591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:17.455749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38150 (local address 127.8.158.125:38439)
0929 19:07:17.455786 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:17.455788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:17.455920 (+ 132us) server_negotiation.cc:407] Connection header received
0929 19:07:17.455977 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:17.455978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:17.455988 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:17.456037 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:17.456334 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:17.456996 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:17.457995 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:17.458287 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:17.458364 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:17.460227 (+ 1863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:17.460228 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:17.460229 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:07:17.460236 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:17.461471 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:17.461778 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:17.461779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:17.461779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:17.461830 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:17.462044 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:17.462045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:17.462045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:17.462215 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:17.462276 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:17.462454 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 19:07:17.462528 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 19:07:17.463092 31661 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.158.124" port: 34097 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 19:07:17 dist-test-slave-nmsz krb5kdc[31430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.158.126@KRBTEST.COM
W20260929 19:07:17.465716 31679 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5078132eba2d40659afc2b299bad79ff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 19:07:17.465785 31679 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:07:17.464255 31668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5078132eba2d40659afc2b299bad79ff [sys.catalog]: This master's current role is: FOLLOWER
Sep 29 19:07:17 dist-test-slave-nmsz krb5kdc[31430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708837, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.124@KRBTEST.COM for kudu/127.8.158.124@KRBTEST.COM
I20260929 19:07:17.465044 31518 master_runner.cc:186] Error getting master registration for 127.8.158.126:37381: OK, instance_id { permanent_uuid: "3e45422dfed440bc9bff6bedc922a306" instance_seqno: 1790708837151209 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 19:07:17.468967 31518 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 19:07:17.469264 31514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:17.458379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37550 (local address 127.8.158.126:37381)
0929 19:07:17.458417 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:17.458419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:17.458430 (+ 11us) server_negotiation.cc:407] Connection header received
0929 19:07:17.458512 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:17.458514 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:17.458523 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:17.458578 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:17.459151 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:17.459680 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:17.462999 (+ 3319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:17.463270 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:17.463338 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:17.466217 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:17.466218 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:17.466219 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:07:17.466226 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:17.467895 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:17.468252 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:17.468254 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:17.468254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:17.468318 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:17.468546 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:17.468547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:17.468547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:17.468665 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:17.468771 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:17.469104 (+ 333us) server_negotiation.cc:299] Negotiation successful
0929 19:07:17.469181 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 19:07:17.469986 31663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:17.463287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.124:34097 (local address 127.0.0.1:59934)
0929 19:07:17.463333 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 19:07:17.463344 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:07:17.463386 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:07:17.463533 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:07:17.463534 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:07:17.463647 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:07:17.463694 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:07:17.463695 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:07:17.464401 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:07:17.464402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:07:17.465024 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:07:17.465027 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:07:17.465456 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:07:17.465456 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:07:17.465500 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:17.465910 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:07:17.465915 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:07:17.467248 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:07:17.469072 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:07:17.469072 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:07:17.469073 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:07:17.469226 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:07:17.469415 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:07:17.469416 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:07:17.469416 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:07:17.469467 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:07:17.469693 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:07:17.469694 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:07:17.469728 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 19:07:17.469791 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 19:07:17.469883 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 19:07:17.470043 31666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:17.463362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59934 (local address 127.8.158.124:34097)
0929 19:07:17.463405 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:17.463407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:17.463413 (+ 6us) server_negotiation.cc:407] Connection header received
0929 19:07:17.463433 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:17.463433 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:17.463446 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:17.463754 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:17.463774 (+ 20us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:17.464312 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:17.465106 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:17.465384 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:17.465633 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:17.467335 (+ 1702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:17.467335 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:17.467336 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:07:17.467343 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:17.468997 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:17.469302 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:17.469302 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:17.469303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:17.469349 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:17.469532 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:17.469534 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:17.469534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:17.469632 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:17.469701 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:17.469832 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 19:07:17.469930 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":28}
I20260929 19:07:17.471161 31661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2f827e6065a4104b2f63037514cafb1: Bootstrap starting.
I20260929 19:07:17.471680 31661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2f827e6065a4104b2f63037514cafb1: Neither blocks nor log segments found. Creating new log.
I20260929 19:07:17.471946 31661 log.cc:824] T 00000000000000000000000000000000 P d2f827e6065a4104b2f63037514cafb1: Log is configured to *not* fsync() on all Append() calls
I20260929 19:07:17.472652 31661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2f827e6065a4104b2f63037514cafb1: No bootstrap required, opened a new log
I20260929 19:07:17.474655 31661 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2f827e6065a4104b2f63037514cafb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e45422dfed440bc9bff6bedc922a306" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37381 } } peers { permanent_uuid: "5078132eba2d40659afc2b299bad79ff" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 38439 } } peers { permanent_uuid: "d2f827e6065a4104b2f63037514cafb1" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 34097 } }
I20260929 19:07:17.474798 31661 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2f827e6065a4104b2f63037514cafb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2f827e6065a4104b2f63037514cafb1, State: Initialized, Role: FOLLOWER
I20260929 19:07:17.474884 31661 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2f827e6065a4104b2f63037514cafb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e45422dfed440bc9bff6bedc922a306" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37381 } } peers { permanent_uuid: "5078132eba2d40659afc2b299bad79ff" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 38439 } } peers { permanent_uuid: "d2f827e6065a4104b2f63037514cafb1" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 34097 } }
I20260929 19:07:17.475157 31661 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2f827e6065a4104b2f63037514cafb1 [sys.catalog]: configured and running, proceeding with master startup.
Sep 29 19:07:17 dist-test-slave-nmsz krb5kdc[31430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.158.125@KRBTEST.COM
I20260929 19:07:17.476531 31682 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2f827e6065a4104b2f63037514cafb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e45422dfed440bc9bff6bedc922a306" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37381 } } peers { permanent_uuid: "5078132eba2d40659afc2b299bad79ff" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 38439 } } peers { permanent_uuid: "d2f827e6065a4104b2f63037514cafb1" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 34097 } } }
I20260929 19:07:17.476863 31682 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2f827e6065a4104b2f63037514cafb1 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 19:07:17.477427 31693 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d2f827e6065a4104b2f63037514cafb1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 19:07:17.477525 31693 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:07:17.477108 31593 master_runner.cc:186] Error getting master registration for 127.8.158.126:37381: OK, instance_id { permanent_uuid: "3e45422dfed440bc9bff6bedc922a306" instance_seqno: 1790708837151209 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 19:07:17.479588 31593 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 19:07:17.479779 31591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:17.472177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38154 (local address 127.8.158.125:38439)
0929 19:07:17.472225 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:17.472227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:17.472235 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:07:17.472253 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:17.472254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:17.472262 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:17.472337 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:17.472806 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:17.473213 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:17.473890 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:17.474068 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:17.474157 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:17.476525 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:17.476526 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:17.476527 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:07:17.476533 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:17.478273 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:17.478705 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:17.478706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:17.478706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:17.478759 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:17.478942 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:17.478943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:17.478943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:17.479058 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:17.479177 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:17.479624 (+ 447us) server_negotiation.cc:299] Negotiation successful
0929 19:07:17.479699 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 19:07:17.481097 8825 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:07:17.482134 31485 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "3e45422dfed440bc9bff6bedc922a306"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37550
I20260929 19:07:17.493682 31511 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.158.124" port: 34097 } attempt: 2
Sep 29 19:07:17 dist-test-slave-nmsz krb5kdc[31430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708837, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.158.126@KRBTEST.COM for kudu/127.8.158.124@KRBTEST.COM
I20260929 19:07:17.501497 31513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:17.493923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.158.124:34097 (local address 127.0.0.1:59938)
0929 19:07:17.493966 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0929 19:07:17.493976 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:07:17.494011 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:07:17.494298 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:07:17.494299 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:07:17.494482 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:07:17.494543 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:07:17.494545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:07:17.495436 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:07:17.495437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:07:17.496243 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:07:17.496245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:07:17.496754 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:07:17.496755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:07:17.496814 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:17.497260 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:07:17.497265 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:07:17.498687 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:07:17.500455 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:07:17.500457 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:07:17.500457 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:07:17.500677 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:07:17.500874 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:07:17.500875 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:07:17.500875 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:07:17.500939 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:07:17.501188 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:07:17.501189 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:07:17.501233 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 19:07:17.501302 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 19:07:17.501403 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260929 19:07:17.501816 31666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:17.494105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59938 (local address 127.8.158.124:34097)
0929 19:07:17.494146 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:17.494147 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:17.494174 (+ 27us) server_negotiation.cc:407] Connection header received
0929 19:07:17.494194 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:17.494195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:17.494205 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:17.494266 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:17.494627 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:17.495341 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:17.496331 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:17.496634 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:17.496701 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:17.498763 (+ 2062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:17.498764 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:17.498765 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:07:17.498772 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:17.500371 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:17.500745 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:17.500746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:17.500747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:17.500809 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:17.501004 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:17.501005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:17.501005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:17.501125 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:17.501187 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:17.501656 (+ 469us) server_negotiation.cc:299] Negotiation successful
0929 19:07:17.501728 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 19:07:17.503243 31511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e45422dfed440bc9bff6bedc922a306: Bootstrap starting.
I20260929 19:07:17.503929 31511 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e45422dfed440bc9bff6bedc922a306: Neither blocks nor log segments found. Creating new log.
I20260929 19:07:17.504199 31511 log.cc:824] T 00000000000000000000000000000000 P 3e45422dfed440bc9bff6bedc922a306: Log is configured to *not* fsync() on all Append() calls
I20260929 19:07:17.505019 31511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e45422dfed440bc9bff6bedc922a306: No bootstrap required, opened a new log
I20260929 19:07:17.506354 31511 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e45422dfed440bc9bff6bedc922a306 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e45422dfed440bc9bff6bedc922a306" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37381 } } peers { permanent_uuid: "5078132eba2d40659afc2b299bad79ff" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 38439 } } peers { permanent_uuid: "d2f827e6065a4104b2f63037514cafb1" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 34097 } }
I20260929 19:07:17.506479 31511 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e45422dfed440bc9bff6bedc922a306 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e45422dfed440bc9bff6bedc922a306, State: Initialized, Role: FOLLOWER
I20260929 19:07:17.506546 31511 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e45422dfed440bc9bff6bedc922a306 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e45422dfed440bc9bff6bedc922a306" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37381 } } peers { permanent_uuid: "5078132eba2d40659afc2b299bad79ff" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 38439 } } peers { permanent_uuid: "d2f827e6065a4104b2f63037514cafb1" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 34097 } }
I20260929 19:07:17.506811 31696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e45422dfed440bc9bff6bedc922a306 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e45422dfed440bc9bff6bedc922a306" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37381 } } peers { permanent_uuid: "5078132eba2d40659afc2b299bad79ff" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 38439 } } peers { permanent_uuid: "d2f827e6065a4104b2f63037514cafb1" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 34097 } } }
I20260929 19:07:17.506888 31511 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e45422dfed440bc9bff6bedc922a306 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:07:17.506899 31696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e45422dfed440bc9bff6bedc922a306 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:07:17.508258 31485 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e45422dfed440bc9bff6bedc922a306 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 19:07:17.508359 31485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e45422dfed440bc9bff6bedc922a306 [term 0 FOLLOWER]: Advancing to term 1
W20260929 19:07:17.508596 31707 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3e45422dfed440bc9bff6bedc922a306: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 19:07:17.508653 31707 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:07:17.509245 31485 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e45422dfed440bc9bff6bedc922a306 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e45422dfed440bc9bff6bedc922a306" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37381 } } peers { permanent_uuid: "5078132eba2d40659afc2b299bad79ff" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 38439 } } peers { permanent_uuid: "d2f827e6065a4104b2f63037514cafb1" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 34097 } }
I20260929 19:07:17.509770 31560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e45422dfed440bc9bff6bedc922a306" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5078132eba2d40659afc2b299bad79ff"
I20260929 19:07:17.509913 31560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5078132eba2d40659afc2b299bad79ff [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:07:17.510776 31560 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5078132eba2d40659afc2b299bad79ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e45422dfed440bc9bff6bedc922a306 in term 1.
I20260929 19:07:17.511005 31461 leader_election.cc:302] T 00000000000000000000000000000000 P 3e45422dfed440bc9bff6bedc922a306 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e45422dfed440bc9bff6bedc922a306, 5078132eba2d40659afc2b299bad79ff; no voters:
I20260929 19:07:17.511113 31696 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e45422dfed440bc9bff6bedc922a306 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:07:17.511240 31696 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e45422dfed440bc9bff6bedc922a306 [term 1 LEADER]: Becoming Leader. State: Replica: 3e45422dfed440bc9bff6bedc922a306, State: Running, Role: LEADER
I20260929 19:07:17.511325 31696 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e45422dfed440bc9bff6bedc922a306 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e45422dfed440bc9bff6bedc922a306" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37381 } } peers { permanent_uuid: "5078132eba2d40659afc2b299bad79ff" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 38439 } } peers { permanent_uuid: "d2f827e6065a4104b2f63037514cafb1" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 34097 } }
I20260929 19:07:17.511684 31696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e45422dfed440bc9bff6bedc922a306 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e45422dfed440bc9bff6bedc922a306. Latest consensus state: current_term: 1 leader_uuid: "3e45422dfed440bc9bff6bedc922a306" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e45422dfed440bc9bff6bedc922a306" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37381 } } peers { permanent_uuid: "5078132eba2d40659afc2b299bad79ff" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 38439 } } peers { permanent_uuid: "d2f827e6065a4104b2f63037514cafb1" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 34097 } } }
I20260929 19:07:17.511742 31696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e45422dfed440bc9bff6bedc922a306 [sys.catalog]: This master's current role is: LEADER
I20260929 19:07:17.512269 31635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e45422dfed440bc9bff6bedc922a306" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d2f827e6065a4104b2f63037514cafb1"
I20260929 19:07:17.512372 31635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2f827e6065a4104b2f63037514cafb1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:07:17.513168 31635 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d2f827e6065a4104b2f63037514cafb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e45422dfed440bc9bff6bedc922a306 in term 1.
I20260929 19:07:17.513406 31485 leader_election.cc:288] T 00000000000000000000000000000000 P 3e45422dfed440bc9bff6bedc922a306 [CANDIDATE]: Term 1 election: Requested vote from peers 5078132eba2d40659afc2b299bad79ff (127.8.158.125:38439), d2f827e6065a4104b2f63037514cafb1 (127.8.158.124:34097)
I20260929 19:07:17.514945 31711 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:07:17.515141 31711 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:07:17.517658 31635 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d2f827e6065a4104b2f63037514cafb1 [term 1 FOLLOWER]: Refusing update from remote peer 3e45422dfed440bc9bff6bedc922a306: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:07:17.517915 31560 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5078132eba2d40659afc2b299bad79ff [term 1 FOLLOWER]: Refusing update from remote peer 3e45422dfed440bc9bff6bedc922a306: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:07:17.518404 31696 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3e45422dfed440bc9bff6bedc922a306 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5078132eba2d40659afc2b299bad79ff" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 38439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:07:17.518544 31696 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3e45422dfed440bc9bff6bedc922a306 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2f827e6065a4104b2f63037514cafb1" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 34097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:07:17.519726 31668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5078132eba2d40659afc2b299bad79ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e45422dfed440bc9bff6bedc922a306. Latest consensus state: current_term: 1 leader_uuid: "3e45422dfed440bc9bff6bedc922a306" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e45422dfed440bc9bff6bedc922a306" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37381 } } peers { permanent_uuid: "5078132eba2d40659afc2b299bad79ff" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 38439 } } peers { permanent_uuid: "d2f827e6065a4104b2f63037514cafb1" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 34097 } } }
I20260929 19:07:17.519812 31668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5078132eba2d40659afc2b299bad79ff [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:07:17.520720 31682 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2f827e6065a4104b2f63037514cafb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e45422dfed440bc9bff6bedc922a306. Latest consensus state: current_term: 1 leader_uuid: "3e45422dfed440bc9bff6bedc922a306" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e45422dfed440bc9bff6bedc922a306" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37381 } } peers { permanent_uuid: "5078132eba2d40659afc2b299bad79ff" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 38439 } } peers { permanent_uuid: "d2f827e6065a4104b2f63037514cafb1" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 34097 } } }
I20260929 19:07:17.520805 31682 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2f827e6065a4104b2f63037514cafb1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:07:17.531173 31514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:17.515992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37554 (local address 127.8.158.126:37381)
0929 19:07:17.516056 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:17.516058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:17.517760 (+ 1702us) server_negotiation.cc:407] Connection header received
0929 19:07:17.518059 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:17.518060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:17.518070 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:17.518098 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:17.520909 (+ 2811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:17.521592 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:17.524190 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:17.524478 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:17.524516 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:17.526588 (+ 2072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:17.526589 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:17.526589 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:07:17.526595 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:17.528262 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:17.529652 (+ 1390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:17.529653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:17.529654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:17.529705 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:17.530201 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:17.530202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:17.530202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:17.530349 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:17.530430 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:17.531011 (+ 581us) server_negotiation.cc:299] Negotiation successful
0929 19:07:17.531085 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 19:07:17.531606 31696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e45422dfed440bc9bff6bedc922a306 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e45422dfed440bc9bff6bedc922a306" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e45422dfed440bc9bff6bedc922a306" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37381 } } peers { permanent_uuid: "5078132eba2d40659afc2b299bad79ff" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 38439 } } peers { permanent_uuid: "d2f827e6065a4104b2f63037514cafb1" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 34097 } } }
I20260929 19:07:17.531692 31696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e45422dfed440bc9bff6bedc922a306 [sys.catalog]: This master's current role is: LEADER
I20260929 19:07:17.531778 31716 mvcc.cc:205] Tried to move back new op lower bound from 7334743398470221824 to 7334743398447013888. Current Snapshot: MvccSnapshot[applied={T|T < 7334743398470221824}]
I20260929 19:07:17.532378 31682 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2f827e6065a4104b2f63037514cafb1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e45422dfed440bc9bff6bedc922a306" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e45422dfed440bc9bff6bedc922a306" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37381 } } peers { permanent_uuid: "5078132eba2d40659afc2b299bad79ff" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 38439 } } peers { permanent_uuid: "d2f827e6065a4104b2f63037514cafb1" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 34097 } } }
I20260929 19:07:17.532457 31682 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2f827e6065a4104b2f63037514cafb1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:07:17.532653 31722 mvcc.cc:205] Tried to move back new op lower bound from 7334743398470221824 to 7334743398447013888. Current Snapshot: MvccSnapshot[applied={T|T < 7334743398470221824}]
I20260929 19:07:17.532832 31696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e45422dfed440bc9bff6bedc922a306 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e45422dfed440bc9bff6bedc922a306" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e45422dfed440bc9bff6bedc922a306" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37381 } } peers { permanent_uuid: "5078132eba2d40659afc2b299bad79ff" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 38439 } } peers { permanent_uuid: "d2f827e6065a4104b2f63037514cafb1" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 34097 } } }
I20260929 19:07:17.532922 31696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e45422dfed440bc9bff6bedc922a306 [sys.catalog]: This master's current role is: LEADER
I20260929 19:07:17.532989 31666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:17.516759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59950 (local address 127.8.158.124:34097)
0929 19:07:17.516793 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:17.516795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:17.517958 (+ 1163us) server_negotiation.cc:407] Connection header received
0929 19:07:17.518840 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:17.518841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:17.518852 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:17.518885 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:17.519951 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:17.520657 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:17.523535 (+ 2878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:17.523833 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:17.523874 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:17.526864 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:17.526865 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:17.526865 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:07:17.526872 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:17.529027 (+ 2155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:17.529892 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:17.529892 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:17.529893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:17.529940 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:17.530598 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:17.530599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:17.530599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:17.530711 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:17.530776 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:17.532833 (+ 2057us) server_negotiation.cc:299] Negotiation successful
0929 19:07:17.532904 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 19:07:17.533200 31668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5078132eba2d40659afc2b299bad79ff [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e45422dfed440bc9bff6bedc922a306" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e45422dfed440bc9bff6bedc922a306" member_type: VOTER last_known_addr { host: "127.8.158.126" port: 37381 } } peers { permanent_uuid: "5078132eba2d40659afc2b299bad79ff" member_type: VOTER last_known_addr { host: "127.8.158.125" port: 38439 } } peers { permanent_uuid: "d2f827e6065a4104b2f63037514cafb1" member_type: VOTER last_known_addr { host: "127.8.158.124" port: 34097 } } }
I20260929 19:07:17.533299 31668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5078132eba2d40659afc2b299bad79ff [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:07:17.534650 31720 mvcc.cc:205] Tried to move back new op lower bound from 7334743398470221824 to 7334743398447013888. Current Snapshot: MvccSnapshot[applied={T|T < 7334743398470221824}]
I20260929 19:07:17.535327 31591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:17.516690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38168 (local address 127.8.158.125:38439)
0929 19:07:17.516728 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:17.516730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:17.518038 (+ 1308us) server_negotiation.cc:407] Connection header received
0929 19:07:17.518061 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:17.518062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:17.518073 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:17.518131 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:17.526391 (+ 8260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:17.527109 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:17.528140 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:17.528437 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:17.528476 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:17.531930 (+ 3454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:07:17.531932 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:07:17.531932 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:07:17.531939 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:07:17.534072 (+ 2133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:17.534543 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:17.534544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:17.534544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:17.534599 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:07:17.534785 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:07:17.534786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:07:17.534786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:07:17.534916 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:07:17.534989 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:17.535170 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 19:07:17.535242 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 19:07:17.535866 31711 catalog_manager.cc:1347] Generated new cluster ID: 41f69257478247dfa5723ead696c071d
I20260929 19:07:17.535912 31711 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:07:17.563963 31711 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:07:17.564819 31711 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:07:17.574397 31711 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e45422dfed440bc9bff6bedc922a306: Generated new TSK 0
I20260929 19:07:17.574640 31711 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 19:07:17.599532 31591 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:07:17.598008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38178 (local address 127.8.158.125:38439)
0929 19:07:17.598049 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:17.598051 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:17.598919 (+ 868us) server_negotiation.cc:407] Connection header received
0929 19:07:17.599016 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:17.599017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:17.599029 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:17.599059 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:17.599469 (+ 410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38178: BlockingRecv error: recv got EOF from 127.0.0.1:38178 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260929 19:07:17.600529 31514 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:07:17.598375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37564 (local address 127.8.158.126:37381)
0929 19:07:17.598418 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:17.598420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:17.600011 (+ 1591us) server_negotiation.cc:407] Connection header received
0929 19:07:17.600105 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:17.600106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:17.600116 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:17.600145 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:17.600459 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37564: BlockingRecv error: recv got EOF from 127.0.0.1:37564 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260929 19:07:17.601429 31666 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:07:17.598649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59958 (local address 127.8.158.124:34097)
0929 19:07:17.598686 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:17.598688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:17.600862 (+ 2174us) server_negotiation.cc:407] Connection header received
0929 19:07:17.601024 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:17.601025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:17.601035 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:17.601064 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:17.601375 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59958: BlockingRecv error: recv got EOF from 127.0.0.1:59958 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260929 19:07:17.603722 31591 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:07:17.602992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38186 (local address 127.8.158.125:38439)
0929 19:07:17.603023 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:17.603025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:17.603242 (+ 217us) server_negotiation.cc:407] Connection header received
0929 19:07:17.603338 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:17.603339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:17.603355 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:17.603382 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:17.603678 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38186: BlockingRecv error: recv got EOF from 127.0.0.1:38186 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260929 19:07:17.606678 31591 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:07:17.605897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38200 (local address 127.8.158.125:38439)
0929 19:07:17.605928 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:17.605930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:17.606152 (+ 222us) server_negotiation.cc:407] Connection header received
0929 19:07:17.606244 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:17.606245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:17.606255 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:17.606281 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:17.606635 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38200: BlockingRecv error: recv got EOF from 127.0.0.1:38200 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260929 19:07:17.610476 31591 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:07:17.609752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38214 (local address 127.8.158.125:38439)
0929 19:07:17.609782 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:17.609785 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:17.609990 (+ 205us) server_negotiation.cc:407] Connection header received
0929 19:07:17.610081 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:17.610082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:17.610091 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:17.610119 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:17.610438 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38214: BlockingRecv error: recv got EOF from 127.0.0.1:38214 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260929 19:07:17.616310 31591 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:07:17.615583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38218 (local address 127.8.158.125:38439)
0929 19:07:17.615613 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:17.615615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:17.615837 (+ 222us) server_negotiation.cc:407] Connection header received
0929 19:07:17.615932 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:17.615933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:17.615942 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:17.615970 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:17.616272 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38218: BlockingRecv error: recv got EOF from 127.0.0.1:38218 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260929 19:07:17.626151 31591 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:07:17.625403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38230 (local address 127.8.158.125:38439)
0929 19:07:17.625435 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:17.625437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:17.625655 (+ 218us) server_negotiation.cc:407] Connection header received
0929 19:07:17.625749 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:17.625750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:17.625759 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:17.625785 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:17.626105 (+ 320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38230: BlockingRecv error: recv got EOF from 127.0.0.1:38230 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260929 19:07:17.644052 31591 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:07:17.643353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38234 (local address 127.8.158.125:38439)
0929 19:07:17.643393 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:17.643396 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:17.643530 (+ 134us) server_negotiation.cc:407] Connection header received
0929 19:07:17.643599 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:17.643600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:17.643611 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:17.643680 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:17.643985 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38234: BlockingRecv error: recv got EOF from 127.0.0.1:38234 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260929 19:07:17.677783 31591 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:07:17.677074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38240 (local address 127.8.158.125:38439)
0929 19:07:17.677122 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:17.677125 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:17.677257 (+ 132us) server_negotiation.cc:407] Connection header received
0929 19:07:17.677334 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:17.677335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:17.677348 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:17.677416 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:17.677724 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38240: BlockingRecv error: recv got EOF from 127.0.0.1:38240 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260929 19:07:17.743525 31591 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:07:17.742854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38252 (local address 127.8.158.125:38439)
0929 19:07:17.742944 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:17.742946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:17.742960 (+ 14us) server_negotiation.cc:407] Connection header received
0929 19:07:17.743044 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:17.743045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:17.743056 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:17.743109 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:17.743467 (+ 358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38252: BlockingRecv error: recv got EOF from 127.0.0.1:38252 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 19:07:17.873247 31591 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:07:17.872586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38266 (local address 127.8.158.125:38439)
0929 19:07:17.872680 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:17.872682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:17.872730 (+ 48us) server_negotiation.cc:407] Connection header received
0929 19:07:17.872792 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:17.872792 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:17.872803 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:17.872843 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:17.873173 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38266: BlockingRecv error: recv got EOF from 127.0.0.1:38266 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260929 19:07:18.131063 31591 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:07:18.130371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38280 (local address 127.8.158.125:38439)
0929 19:07:18.130429 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:18.130432 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:18.130465 (+ 33us) server_negotiation.cc:407] Connection header received
0929 19:07:18.130560 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:18.130561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:18.130574 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:18.130646 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:07:18.130973 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38280: BlockingRecv error: recv got EOF from 127.0.0.1:38280 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 19:07:18.466185 31679 catalog_manager.cc:1259] Loaded cluster ID: 41f69257478247dfa5723ead696c071d
I20260929 19:07:18.466269 31679 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5078132eba2d40659afc2b299bad79ff: loading cluster ID for follower catalog manager: success
I20260929 19:07:18.467156 31679 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5078132eba2d40659afc2b299bad79ff: acquiring CA information for follower catalog manager: success
I20260929 19:07:18.467438 31679 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5078132eba2d40659afc2b299bad79ff: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 19:07:18.477892 31693 catalog_manager.cc:1259] Loaded cluster ID: 41f69257478247dfa5723ead696c071d
I20260929 19:07:18.477962 31693 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d2f827e6065a4104b2f63037514cafb1: loading cluster ID for follower catalog manager: success
I20260929 19:07:18.478704 31693 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d2f827e6065a4104b2f63037514cafb1: acquiring CA information for follower catalog manager: success
I20260929 19:07:18.478921 31693 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d2f827e6065a4104b2f63037514cafb1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 19:07:18.646548 31792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:18.644014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38294 (local address 127.8.158.125:38439)
0929 19:07:18.644138 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:18.644140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:18.644151 (+ 11us) server_negotiation.cc:407] Connection header received
0929 19:07:18.644211 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:18.644211 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:18.644254 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:18.644329 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:07:18.644474 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:18.644992 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:18.645737 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:18.645945 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:18.646011 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:18.646177 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:07:18.646290 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:07:18.646414 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:18.646463 (+ 49us) server_negotiation.cc:299] Negotiation successful
0929 19:07:18.646483 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":70,"threads_started":1}
I20260929 19:07:18.650439 31798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:18.648196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59972 (local address 127.8.158.124:34097)
0929 19:07:18.648302 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:18.648304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:18.648312 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:07:18.648366 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:18.648367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:18.648398 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:18.648462 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:07:18.648523 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:18.648984 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:18.649634 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:18.649857 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:18.649917 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:18.650059 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:07:18.650169 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:07:18.650236 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:18.650277 (+ 41us) server_negotiation.cc:299] Negotiation successful
0929 19:07:18.650296 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":53,"threads_started":1}
I20260929 19:07:18.654677 31792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:18.651926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38300 (local address 127.8.158.125:38439)
0929 19:07:18.651968 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:18.651970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:18.652066 (+ 96us) server_negotiation.cc:407] Connection header received
0929 19:07:18.652172 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:18.652173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:18.652184 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:18.652213 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:07:18.652358 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:18.653112 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:18.653934 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:18.654198 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:18.654258 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:18.654402 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:07:18.654434 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:07:18.654480 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:18.654573 (+ 93us) server_negotiation.cc:299] Negotiation successful
0929 19:07:18.654594 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 19:07:18.655113 31805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:18.651982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37572 (local address 127.8.158.126:37381)
0929 19:07:18.652200 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:18.652202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:18.652210 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:07:18.652248 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:18.652253 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:18.652291 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:18.652385 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:07:18.652520 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:18.653080 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:18.654079 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:18.654485 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:18.654542 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:18.654690 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:07:18.654798 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:07:18.654851 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:18.655009 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 19:07:18.655032 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":63,"threads_started":1}
I20260929 19:07:18.655223 31798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:07:18.652137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59988 (local address 127.8.158.124:34097)
0929 19:07:18.652174 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 19:07:18.652176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:07:18.652464 (+ 288us) server_negotiation.cc:407] Connection header received
0929 19:07:18.652524 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:07:18.652525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:07:18.652535 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:07:18.652589 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:07:18.652701 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:18.653385 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:18.654354 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:07:18.654699 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:07:18.654757 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:07:18.654932 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:07:18.654965 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:07:18.655013 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 19:07:18.655084 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 19:07:18.655105 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 19:07:18.660832 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 31443
I20260929 19:07:18.665890 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 31518
I20260929 19:07:18.670586 8825 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkdiMiM/build/release/bin/kudu with pid 31593
2026-09-29T19:07:18Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2770 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5571 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (264940 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST